Skip to content

Commit 8908b07

Browse files
committed
Add process-bound environments documentation
1 parent 0663009 commit 8908b07

File tree

2 files changed

+141
-0
lines changed

2 files changed

+141
-0
lines changed

CHANGELOG.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,30 @@
11
# Changelog
22

3+
## 2.2.0 (2026-03-13)
4+
5+
### Added
6+
7+
- **Process-Bound Python Environments** - Each Erlang process gets an isolated Python namespace
8+
- `py:get_local_env/1` - Get or create a process-local Python environment
9+
- Variables defined via `py:exec()` persist across calls within the same Erlang process
10+
- Automatic cleanup when the Erlang process exits (no manual deallocation needed)
11+
- Environments are stored in process dictionary, keyed by interpreter ID
12+
- Works with both subinterpreter and worker modes
13+
- Critical for process affinity: each process maintains independent Python state
14+
- Memory-safe: environments created inside the correct interpreter's allocator
15+
- See [Process-Bound Environments](docs/context-affinity.md#process-bound-environments) for details
16+
17+
- **Docker Test Configs** - Containerized test environment
18+
- `docker/Dockerfile.python312` - Python 3.12 test image
19+
- `docker/Dockerfile.python314` - Python 3.14 test image
20+
- `docker/Dockerfile.asan` - AddressSanitizer build for memory testing
21+
- `docker/docker-compose.yml` - Multi-container test orchestration
22+
- `docker/run-tests.sh` - Automated test runner script
23+
24+
- **Async Task Benchmark** - Performance testing for async operations
25+
- `examples/bench_async_task.erl` - Erlang benchmark runner
26+
- `priv/test_async_task.py` - Python async task implementation
27+
328
## 2.1.0 (2026-03-12)
429

530
### Added

docs/context-affinity.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,10 +240,126 @@ case py:contexts_started() of
240240
end.
241241
```
242242

243+
## Process-Bound Environments
244+
245+
Process-bound environments provide true process-level isolation for Python state. Each Erlang process automatically gets its own Python namespace that persists across calls.
246+
247+
### How It Works
248+
249+
When you call `py:call()`, `py:eval()`, or `py:exec()`, the library automatically:
250+
251+
1. Looks up or creates a process-local Python environment for your Erlang process
252+
2. Executes the Python code using that environment
253+
3. Stores variables, imports, and objects in that environment
254+
4. Cleans up automatically when your Erlang process exits
255+
256+
This happens transparently - no explicit binding required.
257+
258+
### Basic Usage
259+
260+
```erlang
261+
%% Get a context
262+
Ctx = py:context(1),
263+
264+
%% Define a variable - it persists for THIS Erlang process
265+
ok = py:exec(Ctx, <<"counter = 0">>),
266+
ok = py:exec(Ctx, <<"counter += 1">>),
267+
{ok, 1} = py:eval(Ctx, <<"counter">>).
268+
269+
%% In a different Erlang process, counter is independent:
270+
spawn(fun() ->
271+
ok = py:exec(Ctx, <<"counter = 100">>),
272+
{ok, 100} = py:eval(Ctx, <<"counter">>)
273+
end).
274+
275+
%% Back in original process, still 1
276+
{ok, 1} = py:eval(Ctx, <<"counter">>).
277+
```
278+
279+
### Process Affinity for AI Workloads
280+
281+
Process-bound environments are ideal for scenarios where each Erlang process needs isolated Python state:
282+
283+
```erlang
284+
%% Each user session gets its own chat history
285+
handle_user_session(UserId) ->
286+
Ctx = py:context(),
287+
%% Initialize conversation for this process
288+
ok = py:exec(Ctx, <<"
289+
conversation_history = []
290+
def add_message(role, content):
291+
conversation_history.append({'role': role, 'content': content})
292+
def get_history():
293+
return conversation_history
294+
">>),
295+
session_loop(Ctx).
296+
297+
session_loop(Ctx) ->
298+
receive
299+
{user_message, Msg} ->
300+
py:call(Ctx, '__main__', add_message, [<<"user">>, Msg]),
301+
%% Process with AI...
302+
session_loop(Ctx);
303+
get_history ->
304+
{ok, History} = py:call(Ctx, '__main__', get_history, []),
305+
History
306+
end.
307+
```
308+
309+
### Isolation Between Processes
310+
311+
```erlang
312+
%% Process A
313+
spawn(fun() ->
314+
Ctx = py:context(1),
315+
ok = py:exec(Ctx, <<"x = 'from process A'">>)
316+
end),
317+
318+
%% Process B - same context, but isolated environment
319+
spawn(fun() ->
320+
Ctx = py:context(1), %% Same context!
321+
ok = py:exec(Ctx, <<"x = 'from process B'">>),
322+
{ok, <<"from process B">>} = py:eval(Ctx, <<"x">>) %% Own value
323+
end).
324+
```
325+
326+
### Memory Management
327+
328+
Environments are automatically freed when:
329+
- The Erlang process exits (normal, abnormal, or killed)
330+
- The NIF resource destructor runs during garbage collection
331+
332+
No manual cleanup is needed. The environments use the correct memory allocator for each interpreter (critical for subinterpreters which have isolated allocators).
333+
334+
### When to Use Process-Bound Environments
335+
336+
**Good use cases:**
337+
- Stateful sessions (chat, game state, user preferences)
338+
- Long-running workers that accumulate state
339+
- Process-per-request patterns with state
340+
- AI pipelines with per-request context
341+
342+
**Consider alternatives when:**
343+
- State must be shared between Erlang processes (use shared state API instead)
344+
- State needs to outlive the Erlang process (use explicit storage)
345+
- You need multiple independent namespaces per process (use explicit contexts)
346+
347+
### Technical Details
348+
349+
Process-bound environments work by:
350+
351+
1. Storing a `reference()` in the calling process's dictionary under `py_local_env`
352+
2. The reference points to a Python dict created inside the interpreter
353+
3. Each interpreter ID maps to a separate environment (for subinterpreter support)
354+
4. The NIF uses this dict as `locals` for `exec()` and `eval()` operations
355+
356+
For subinterpreters, environments are created inside the target interpreter to ensure memory safety - Python's subinterpreters have isolated memory allocators.
357+
243358
## Best Practices
244359

245360
1. **Use explicit contexts for stateful operations**: `Ctx = py:context(1)` ensures state persists
246361
2. **Use automatic routing for stateless calls**: Let the router handle distribution
247362
3. **Always unbind in finally blocks**: Prevent context leaks
248363
4. **Minimize binding time**: Don't hold contexts longer than necessary
249364
5. **Monitor pool size**: Check `py_context_router:num_contexts()` to understand capacity
365+
6. **Leverage process-bound environments**: For per-process state, rely on automatic environment isolation rather than manual binding

0 commit comments

Comments
 (0)