Thread pool limits meaning is incommensurable across libraries and APIs

[itamarst]

threadpoolctl limiting API has inconsistent behavior

1. Scope of setting limits is inconsistent

threadpoolctl's API to set limits affect different libraries in different ways:

Library	Where is the limit set
OpenMP (libgomp/libomp)	Current thread only
OpenBLAS (pthreads)	Process-wide
OpenBLAS (Windows)	Process-wide
OpenBLAS (OpenMP)	Intended to be process-wide? But currently broken: OpenMathLib/OpenBLAS#5808
MKL (Intel threading)	Process-wide (but both available)
BLIS	Process-wide, (but both available)
FlexiBLAS	Depends on the backend!

See #207

2. Semantics of limits are inconsistent

There are also different semantics for limits. Given a limit of L, this limit can be:

• Per-thread: Each thread can have to L sub-worker threads.
• Process-wide: The process will have a global thread pool with L workers.

Library	Limit is—
OpenMP (libgomp/libomp)	Per-thread
OpenBLAS (pthreads)	Process-wide
OpenBLAS (Windows)	Process-wide
OpenBLAS (OpenMP)	Per-thread
MKL (Intel threading)	Per-thread
BLIS	Per-thread, probably
FlexiBLAS	Depends on the backend!

Demonstration

Consider an example where I start a Python thread pool with 10 threads, and then run a BLAS operation in each thread:

import psutil
from concurrent.futures import ThreadPoolExecutor
import numpy as np
import os
import threading
import threadpoolctl
import sys

A = np.ones((10_000_000,))

num_threads = 0
stop = False

def count_threads():
    global num_threads
    process = psutil.Process()
    while not stop:
        num_threads = max(num_threads, process.num_threads())

def blasop(_ignore):
    threadpoolctl.threadpool_limits(
        limits=int(sys.argv[1]), user_api="blas"
    )
    for i in range(100):
        A.dot(A)

threadpoolctl.threadpool_limits(
    limits=int(sys.argv[1]), user_api="blas"
)
threading.Thread(target=count_threads).start()
POOL = ThreadPoolExecutor(10)
list(POOL.map(blasop, range(10)))
stop = True
print(threadpoolctl.threadpool_info())
print("Max threads:", num_threads)

With OpenBLAS pthreads:

$ python nested-blas.py 10
[{'user_api': 'blas', 'internal_api': 'openblas', 'num_threads': 20, 'prefix': 'libscipy_openblas',
'version': '0.3.31.188.0', 'threading_layer': 'pthreads', 'architecture': 'Haswell'}]
Max threads: 23

With MKL (Conda-Forge):

$ python nested-blas.py 10
[{'user_api': 'blas', 'internal_api': 'mkl', 'num_threads': 12, 'prefix': 'libmkl_rt',  
  'version': '2026.0-Product', 'threading_layer': 'intel'},
 {'user_api': 'openmp', 'internal_api': 'openmp', 'num_threads': 12,
   'prefix': 'libomp', 'version': None}]
Max threads: 102

That's a very different configuration outcome!

Also, in an earlier version I forgot to set the thread pool limit in each thread, which is necessary because of the different semantics...

From Python process pools to Python thread pools

If the limit is set on a process which only runs Python code from its main thread, all the variations have the same effect. Because process pools were more common in Python, the current API therefore does behave consistently.

As Python thread pools become more common, the current state of the API is no longer a good one.

Proposal (big picture)

Notice there are three possible API variants for a library:

• Set the size of a process-wide pool.
• Set the size of thread-specific pool, across all threads.
• Set the size of a thread-specific pool, for the current thread only.

(The fourth quadrant, thread-local setting for a process-wide pool, makes no sense semantically.)

I do not think the current API is viable going forward, it is too inconsistent. On the other hand, it is in use. I therefore propose:

1. Deprecating the current limits API, but leaving it with the current semantics.
2. Add a new API where the user explicitly states which of the 3 variants above they are requesting.
   That is, it should be clear to the user exactly what the outcomes will be.
3. Ideally, leave room for other variations.
4. Add an API for querying which APIs variants are available.

Next steps

If this big picture approach is acceptable, the next step would be coming up with use cases (preventing oversaturation in thread pools, process pool, etc), and then designing a suitable API.

[ogrisel]

Thank you very much for conducting this investigation and summarizing the results so clearly, this is really helpful.

I agree it would be good to introduce new API to inspect which limiting semantics are available.

We can also introduce new API where the caller states explicitly what they want and raise an error or a warning when they request limiting semantics that are not fulfillable for a given runtime.

I think we can also keep the current implicit limiting API unchanged and only document more precisely what the heterogeneity because many existing users might not care. For instance, many users might use threadpoolctl from a Python application with a single outer Python thread and I would rather not bother them.

[itamarst]

For existing API/use case... it is true that if a user wants to set limits for the whole process, and won't be using a Python thread pool that then triggers nested thread pools, the current API is fine. And it saves the user needing to understand all the variants, since the variants aren't relevant in this context.

My main issue is the name, which is quite generic. So maybe... come up with a better name for this use case, make current API an alias of that, document the new name for this use case (and document current API just as a backwards compat alias, and see if we can get IDE to not auto-complete it?). And then maybe eventually deprecate the old API. But that way at least new users will use an API that is clear about what it does.

So that suggests three APIs.

Two high-level, use-case focused APIs:

• For a single outer Python thread; basically, the current semantics.
• For Python thread pools, e.g. joblib threading backend: the API sets thread-local per-thread limit if possible, with some sort of documented best effort fallback policy when that's not possible (OpenBLAS pthreads, at minimum). And maybe that fallback is "install our own thread pool" for OpenBLAS pthreads, at least, and that's acceptable because this is a high-level DWIM policy.

A third API is low-level, and library-specific. For each supported library, provide a more specific introspection and limit setting API that is exposing the underlying library's own APIs, warts and all, just with consistent terminology. This then allows users to create their own custom high-level policies if they choose.

[itamarst]

TIL: https://docs.python.org/3/library/warnings.html#warnings.deprecated (new in 3.13 so would need fallback), hopefully LSP implementations and IDEs will start taking this into account.

[ogrisel]

Sounds like a plan, although the third API sounds like a lot of efforts. The first two items should be enough to avoid the most catastrophic oversubscription cases in scikit-learn with the joblib threading backend on free-threading runtimes.

[itamarst]

The third API is more of a ... prerequisite? At minimum you need to document the semantics of each library's APIs, or maybe switch to a slightly different one. But yes will try to do this in steps and try to do it with minimal path to being useful.

[david-cortes-intel]

Comment here: for MKL, "Intel thread" is actually OpenMP, defaulting to IOMP. In my understanding, as far as linux goes, if an OMP runtime like GOMP is loaded before MKL, then MKL will execute under that OMP backend.

Perhaps @vmalia or @mkrainiuk could comment on what would happen with MKL's thread control functions when GOMP is used as backend and OMP threads are modified before or after MKL threads, either locally or globally.

[itamarst]

The MKL bit was partially based on reading the documentation, which is pretty decent, and does indeed refer to OpenMP.

https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/mkl-set-num-threads.html
https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/mkl-set-num-threads-local.html

[vmalia]

@david-cortes-intel These documents below should answer the question about priority of configuration options from OpenMP vs oneMKL. Thanks @itamarst . Try to use the latest version of the documents though.

The MKL bit was partially based on reading the documentation, which is pretty decent, and does indeed refer to OpenMP.

https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/mkl-set-num-threads.html https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-0/mkl-set-num-threads-local.html

As far as the flavor of OpenMP is concerned, oneMKL provides separate libraries for Intel OpenMP (libmkl_intel_thread) and another one for GNU OpenMP (libmkl_gnu_thread). AFAIK conda-sourced oneMKL uses the mkl_rt(single dynamic library) linkage mode, so the threading layer will be set by the environment variable MKL_THREADING_LAYER as described in https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-linux/2026-0/dynamic-select-the-interface-and-threading-layer.html#DYNAMICALLY-SELECTING-THE-INTERFACE-AND-THREADING-LAYER.

OpenMP vs oneMKL configuration option priority is the same irrespective of which OpenMP flavor/vendor is used.

[itamarst]

Designing an API for thread pools

Motivation

On a machine with C cores:

• If you start a thread pool with C threads, each thread should run OpenMP/BLAS code sequentially, typically set via concurrency level of 1.
• If you start a thread pool with e.g. C / 2 threads, than each thread should run OpenMP/BLAS code with 2 threads each. (Process-global thread pools are a problem for this use case, and I'm going to avoid addressing it, since I think it's out of scope for this particular discussion.)

So, the goal here is to set a per-thread limit.

Constraints

Depending on the underlying library, the ability to set per-thread constraints might be per-thread too... or it could be process-wide. So that suggests sometimes the limit can be set on main thread which kicks off the Python thread pool, but sometimes it needs to be set on every thread in the thread pool.

Sketch of a design (from user perspective)

from multiprocessing.pool import ThreadPool
from os import cpu_count

from threadpoolctl import per_thread_limit


def run(n_threads: int):
    # This will set process-wide limits, for libraries that have that sort of API:
    with per_thread_limit(max(1, cpu_count() // n_threads) as limiter:
        # Tell the ThreadPool to run per-thread limiting, for libraries that have that sort of API:
        with ThreadPool(n_threads, initializer=limiter.limit_in_current_thread) as pool:
            # ... do business logic here ...

Note that we have to do both levels, because sometimes (depending on the underlying library's exposed API) you can set the limit for the whole process, and sometimes it has to be set for each thread.

joblib lacks support for initializer functions but could easily grow that functionality. And it could implement the little song-and-dance itself, so users don't have to, as it already does for process pools.

[itamarst]

Do we need a new low-level per-LibController API?

Is it necessary to differentiate on the LibController API between different variants (per-thread limit set on a per thread basis, per-thread limit set on a process wide basis, or process-wide pool size set on a process wide basis)?

As it turns out... no, it's not necessary.

For process pool use case, it's already fine as is.

In the sketch above, for per-thread limits, you just call the API twice: once in per_thread_limit and once in limit_in_current_thread. This may be redundant, or it may be necessary, but it will give the desired effect either way.

Thus a new low-level API could be done some day in the future, or omitted. But even if the LibController API doesn't need to be improved right now, semantics should at least be documented.

Aside: FlexiBLAS makes everything harder more fun

Not needing to come up with a new low-level API helpful because with FlexiBLAS we may not know what the underlying library's semantics are. Maybe one could empirically detect it though by setting a limit in one thread and checking it in another... but then what happens if you switch backends? So you have to do this runtime detection every time you set a limit, which... is possible I guess.

[ogrisel]

semantics should at least be documented.

I agree. And they should also be programmatically inspectable (e.g. for debugging purpose) and we should add some tests to check that inspection results are aligned with the documentation and the actually observed behavior similarly to what is done in the "Demonstration" section of this issue.

[ogrisel]

The proposed API design for the setting per thread limits looks good to be, but we should also design the inspection API before starting the implementation.

@jeremiedbb any opinion/feedback on the above discussion?