Skip to content

Commit 03a904b

Browse files
committed
Document subinterpreter modes and Python version requirements
- Document four execution paths: SHARED_GIL (3.12+), OWN_GIL (3.14+), free-threaded (3.13+), and BEAM processes - Clarify worker mode is now the default - Explain why OWN_GIL requires Python 3.14+ (C extension global state bugs) - Note that SHARED_GIL isolation improves in Python 3.14+ - Update getting-started.md with context mode examples
1 parent 6fd9229 commit 03a904b

File tree

3 files changed

+83
-19
lines changed

3 files changed

+83
-19
lines changed

README.md

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@ erlang_python embeds Python into the BEAM VM, letting you call Python functions,
1515
evaluate expressions, and stream from generators - all without blocking Erlang
1616
schedulers.
1717

18-
**Three paths to parallelism:**
19-
- **Sub-interpreters** (Python 3.12+) - Each interpreter has its own GIL
20-
- **Free-threaded Python** (3.13+) - No GIL at all
18+
**Parallelism options:**
19+
- **Worker mode** (default, recommended) - Works with any Python version. With free-threaded Python (3.13t+), provides true parallelism automatically
20+
- **SHARED_GIL sub-interpreters** (Python 3.12+) - Isolated namespaces, shared GIL (isolation improves in 3.14+)
21+
- **OWN_GIL sub-interpreters** (Python 3.14+) - Each interpreter has its own GIL, true parallelism
2122
- **BEAM processes** - Fan out work across lightweight Erlang processes
2223

2324
Key features:
@@ -313,10 +314,11 @@ Ref = py:async_call(aiohttp, get, [<<"https://api.example.com/data">>]),
313314

314315
## Parallel Execution with Sub-interpreters
315316

316-
True parallelism without GIL contention using Python 3.12+ sub-interpreters:
317+
True parallelism without GIL contention using Python 3.14+ OWN_GIL sub-interpreters:
317318

318319
```erlang
319-
%% Execute multiple calls in parallel across sub-interpreters
320+
%% Execute multiple calls in parallel across OWN_GIL sub-interpreters
321+
%% Requires Python 3.14+
320322
{ok, Results} = py:parallel([
321323
{math, factorial, [100]},
322324
{math, factorial, [200]},
@@ -326,6 +328,8 @@ True parallelism without GIL contention using Python 3.12+ sub-interpreters:
326328
%% Each call runs in its own interpreter with its own GIL
327329
```
328330

331+
For Python 3.12/3.13, use SHARED_GIL sub-interpreters (`mode => subinterp`) for namespace isolation, but note that parallelism is limited by the shared GIL.
332+
329333
## Parallel Processing with BEAM Processes
330334

331335
Leverage Erlang's lightweight processes for massive parallelism:
@@ -595,19 +599,47 @@ ok = py:clear_traces().
595599

596600
## Execution Modes
597601

598-
The library auto-detects the best execution mode:
602+
### Context Modes
599603

600-
| Mode | Python Version | Parallelism |
604+
When creating Python contexts, you can choose the execution mode:
605+
606+
| Mode | Python Version | Description |
601607
|------|----------------|-------------|
602-
| Free-threaded | 3.13+ (nogil) | True parallel, no GIL |
603-
| Sub-interpreter | 3.12+ | Per-interpreter GIL |
604-
| Multi-executor | Any | GIL contention |
608+
| `worker` | Any | Main interpreter, shared namespace (default, recommended) |
609+
| `subinterp` | 3.12+ | SHARED_GIL sub-interpreter, isolated namespace |
610+
| `owngil` | 3.14+ | OWN_GIL sub-interpreter, true parallelism |
611+
612+
```erlang
613+
%% Default: worker mode (recommended)
614+
%% With free-threaded Python (3.13t+), provides true parallelism automatically
615+
{ok, Ctx} = py_context:new(#{}).
616+
617+
%% Explicit subinterpreter with shared GIL (Python 3.12+)
618+
%% Provides namespace isolation but no parallelism
619+
{ok, Ctx} = py_context:new(#{mode => subinterp}).
620+
621+
%% OWN_GIL mode for true parallelism (Python 3.14+ required)
622+
%% Each context runs in its own pthread with independent GIL
623+
{ok, Ctx} = py_context:new(#{mode => owngil}).
624+
```
625+
626+
**Worker mode is recommended** because it works with any Python version and automatically benefits from free-threaded Python (3.13t+) when available.
627+
628+
**Why OWN_GIL requires Python 3.14+**: Some C extensions (e.g., `_decimal`) have global state bugs in sub-interpreters on Python 3.12/3.13. These are fixed in Python 3.14. SHARED_GIL mode works on 3.12+ but with caveats for C extensions with global state.
605629

606-
Check current mode:
630+
### Runtime Detection
631+
632+
Check the current execution mode:
607633
```erlang
608634
py:execution_mode(). %% => free_threaded | subinterp | multi_executor
609635
```
610636

637+
| Mode | Python Version | Parallelism |
638+
|------|----------------|-------------|
639+
| Free-threaded | 3.13+ (nogil) | True parallel, no GIL |
640+
| Sub-interpreter | 3.12+ | Per-interpreter GIL |
641+
| Multi-executor | Any | GIL contention |
642+
611643
## Error Handling
612644

613645
```erlang

docs/getting-started.md

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,35 @@ ok = py:activate_venv(<<"/path/to/venv">>).
262262
ok = py:deactivate_venv().
263263
```
264264

265-
## Execution Mode and Scalability
265+
## Execution Modes and Context Types
266+
267+
### Context Modes
268+
269+
When creating explicit contexts, you can choose different execution modes:
270+
271+
```erlang
272+
%% Worker mode (default, recommended) - main interpreter
273+
%% With free-threaded Python (3.13t+), provides true parallelism automatically
274+
{ok, Ctx} = py_context:new(#{mode => worker}).
275+
276+
%% SHARED_GIL sub-interpreter (Python 3.12+) - isolated namespace
277+
{ok, Ctx} = py_context:new(#{mode => subinterp}).
278+
279+
%% OWN_GIL sub-interpreter (Python 3.14+) - true parallelism
280+
{ok, Ctx} = py_context:new(#{mode => owngil}).
281+
```
282+
283+
| Mode | Python | Description |
284+
|------|--------|-------------|
285+
| `worker` | Any | Main interpreter, shared namespace (default, recommended) |
286+
| `subinterp` | 3.12+ | SHARED_GIL sub-interpreter, isolated namespace |
287+
| `owngil` | 3.14+ | OWN_GIL sub-interpreter, each has own GIL |
288+
289+
**Worker mode is recommended** because it works with any Python version and automatically benefits from free-threaded Python (3.13t+) when available.
290+
291+
**Why OWN_GIL requires Python 3.14+**: C extensions like `_decimal` have global state bugs in sub-interpreters on Python 3.12/3.13. These are fixed in Python 3.14. SHARED_GIL mode works on 3.12+ but some C extensions may have issues.
292+
293+
### Runtime Detection
266294

267295
Check the current execution mode:
268296

docs/owngil_internals.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22

33
## Overview
44

5-
OWN_GIL mode provides true parallel Python execution using Python 3.12+ per-interpreter GIL (`PyInterpreterConfig_OWN_GIL`). Each OWN_GIL context runs in a dedicated pthread with its own subinterpreter and GIL.
5+
OWN_GIL mode provides true parallel Python execution using Python 3.14+ per-interpreter GIL (`PyInterpreterConfig_OWN_GIL`). Each OWN_GIL context runs in a dedicated pthread with its own subinterpreter and GIL.
6+
7+
**Note**: OWN_GIL requires Python 3.14+ due to C extension global state bugs in earlier versions (e.g., `_decimal`). For Python 3.12/3.13, use SHARED_GIL sub-interpreters (`mode => subinterp`) which provide namespace isolation but share the GIL.
68

79
## Quick Start
810

911
```erlang
10-
%% Create an OWN_GIL context (requires Python 3.12+)
12+
%% Create an OWN_GIL context (requires Python 3.14+)
1113
{ok, Ctx} = py_context:start_link(1, owngil),
1214

1315
%% Basic operations work the same as other modes
@@ -79,11 +81,13 @@ All major erlang_python features work with OWN_GIL mode:
7981

8082
## Comparison with Other Modes
8183

82-
| Mode | Thread Model | GIL | Parallelism |
83-
|------|-------------|-----|-------------|
84-
| `worker` | Dirty scheduler | Main interpreter GIL | None |
85-
| `subinterp` | Dirty scheduler | Shared GIL | None (isolated namespaces) |
86-
| `owngil` | Dedicated pthread | Per-interpreter GIL | True parallel |
84+
| Mode | Python Version | Thread Model | GIL | Parallelism |
85+
|------|----------------|--------------|-----|-------------|
86+
| `worker` | Any | Dirty scheduler | Main interpreter GIL | None |
87+
| `subinterp` | 3.12+ | Dirty scheduler | Shared GIL | None (isolated namespaces) |
88+
| `owngil` | 3.14+ | Dedicated pthread | Per-interpreter GIL | True parallel |
89+
90+
**Why version requirements differ**: The `subinterp` mode (SHARED_GIL) works on Python 3.12+ for namespace isolation. However, `owngil` mode requires Python 3.14+ because C extensions like `_decimal` have global state that crashes in OWN_GIL sub-interpreters on earlier versions. Python 3.14 includes fixes for these issues (see [cpython#106078](https://github.com/python/cpython/issues/106078)).
8791

8892
## Key Data Structures
8993

0 commit comments

Comments
 (0)