[BUG] Default docker /dev/shm size too small for tensor parallelism to work

[robfuscator]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Describe the bug

You guys are awesome, thank you so much for your work!

When using the docker compose file provided in the repo, loading a model with tensor parallelism fails. I get a [Errno 28] No space left on device. error (see logs below). I think I've pinpointed it down, it seems to reference to the space available at /dev/shm.

Increasing the shm_size of the compose service to a larger value allowed to me to load the model successfully. The default is 64mb if my research is correct. Setting it to 2gb did not help, but setting it to 16gb did work. I assume it has to be large enough to fit a whole model layer.

Reproduction steps

1. Spin up the API using the docker-compose.yml provided in this repo
2. Load a model via the API with tensor parallelism enabled (I've tried with Doctor-Shotgun/GLM-4.5-Air-exl3_3.14bpw-h6)

Expected behavior

I'd expect the default docker-compose.yml file to work out of the box. I suggest adding a shm_size that works for most setups. Maybe you have some insights into how much space is required. I'd be willing to create a PR to adjust the docker-compose file and maybe add a note to the wiki if you'd like.

Logs

tabbyapi-1  | 2025-10-11 09:52:28.381 INFO:     127.0.0.1:51336 - "POST /v1/model/load 
tabbyapi-1  | HTTP/1.1" 200
tabbyapi-1  | 2025-10-11 09:52:28.811 INFO:     Using backend exllamav3
tabbyapi-1  | 2025-10-11 09:52:28.815 INFO:     exllamav3 version: 0.0.7
tabbyapi-1  | 2025-10-11 09:52:28.816 WARNING:  ExllamaV3 is currently in an alpha state. 
tabbyapi-1  | Please note that all config options may not work.
tabbyapi-1  | 2025-10-11 09:52:31.175 WARNING:  The provided model does not have vision 
tabbyapi-1  | capabilities that are supported by ExllamaV3. Vision input is disabled.
tabbyapi-1  | 2025-10-11 09:52:31.176 WARNING:  Draft model is disabled because a model name 
tabbyapi-1  | wasn't provided. Please check your config.yml!
tabbyapi-1  | 2025-10-11 09:52:31.176 WARNING:  The given cache size (86000) is not a multiple
tabbyapi-1  | of 256.
tabbyapi-1  | 2025-10-11 09:52:31.176 WARNING:  Overriding cache_size with an overestimated 
tabbyapi-1  | value of 86016 tokens.
tabbyapi-1  | 2025-10-11 09:52:31.177 WARNING:  The given cache_size (86016) is less than 2 * 
tabbyapi-1  | max_seq_len and may be too small for requests using CFG. 
tabbyapi-1  | 2025-10-11 09:52:31.177 WARNING:  Ignore this warning if you do not plan on 
tabbyapi-1  | using CFG.
tabbyapi-1  | 2025-10-11 09:52:31.185 INFO:     Attempting to load a prompt template if 
tabbyapi-1  | present.
tabbyapi-1  | 2025-10-11 09:52:31.211 INFO:     Using template "chat_template" for chat 
tabbyapi-1  | completions.
tabbyapi-1  | 2025-10-11 09:52:31.213 INFO:     Loading model: 
tabbyapi-1  | /app/models/GLM-4.5-Air-exl3_3.14bpw-h6
tabbyapi-1  | 2025-10-11 09:52:31.213 INFO:     Loading with tensor parallel
tabbyapi-1  | /opt/venv/lib/python3.12/site-packages/joblib/_multiprocessing_helpers.py:44: UserWarning: [Errno 28] No space left on device.  joblib will operate in serial mode
tabbyapi-1  |   warnings.warn("%s.  joblib will operate in serial mode" % (e,))
tabbyapi-1  | /opt/venv/lib/python3.12/site-packages/joblib/_multiprocessing_helpers.py:44: UserWarning: [Errno 28] No space left on device.  joblib will operate in serial mode
tabbyapi-1  |   warnings.warn("%s.  joblib will operate in serial mode" % (e,))

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[kingbri1]

None of the maintainers use Docker and it's a more community contributed thing. Feel free to make a PR and I'll be happy to take a look at it!

[robfuscator]

The pre-built images are even more appreciated then!

Do you know what /dev/shm is used for exactly? So I'd be able to suggest a reasonable value in the docs / comments of the compose file. If it's too much of a hassle to find out, I'll just add a note that increasing it might help.

[turboderp]

/dev/shm is the shared memory segment, used for memory allocations shared between processes. It's essential for TP and needs to be a lot more than 64 MB (not sure exactly how much, off the top of my head). But there should be no downside that I can find to just using a much larger size, like 1 GB or more. It's not allocated up-front, only as needed.

[robfuscator]

Thanks for the response! I'll update the docs.

[turboderp]

Just checking the source now, and it looks like exl3 needs a little over 128 MB. However /dev/shm is used by other processes as well, so probably just set it at 1 GB.

[robfuscator]

Odd, at least for me 2GB were not enough. But it should be constant then and not vary by model?

Also, I've noticed that I can't create pull requests for wiki pages, how do you manage the wiki?

[frenzybiscuit]

This fixes behemoth.

It does -not- fix other models.

tabbyapi-1  | Loading model modules                                           1%  1/83 -:--:--
tabbyapi-1  | Traceback (most recent call last):
tabbyapi-1  |   File "/app/main.py", line 181, in <module>
tabbyapi-1  |     entrypoint()
tabbyapi-1  |   File "/app/main.py", line 177, in entrypoint
tabbyapi-1  |     asyncio.run(entrypoint_async())
tabbyapi-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
tabbyapi-1  |     return runner.run(main)
tabbyapi-1  |            ^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
tabbyapi-1  |     return self._loop.run_until_complete(task)
tabbyapi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
tabbyapi-1  |   File "/app/main.py", line 61, in entrypoint_async
tabbyapi-1  |     await model.load_model(
tabbyapi-1  |   File "/app/common/model.py", line 229, in load_model
tabbyapi-1  |     async for _ in load_model_gen(model_path, **kwargs):
tabbyapi-1  |   File "/app/common/model.py", line 205, in load_model_gen
tabbyapi-1  |     async for module, modules in load_status:
tabbyapi-1  |   File "/app/backends/exllamav3/model.py", line 441, in load_gen
tabbyapi-1  |     async for value in iterate_in_threadpool(generator):
tabbyapi-1  |   File "/app/common/concurrency.py", line 30, in iterate_in_threadpool
tabbyapi-1  |     yield await asyncio.to_thread(gen_next, generator)
tabbyapi-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
tabbyapi-1  |     return await loop.run_in_executor(None, func_call)
tabbyapi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
tabbyapi-1  |     result = self.fn(*self.args, **self.kwargs)
tabbyapi-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/app/common/concurrency.py", line 20, in gen_next
tabbyapi-1  |     return next(generator)
tabbyapi-1  |            ^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 59, in generator_context
tabbyapi-1  |     response = gen.send(request)
tabbyapi-1  |                ^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/app/backends/exllamav3/model.py", line 488, in load_model_sync
tabbyapi-1  |     for value in self.model.load_gen(
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/model/model.py", line 310, in load_gen
tabbyapi-1  |     yield from self._load_tp(
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/model/model_tp.py", line 310, in _load_tp
tabbyapi-1  |     exported = module.tp_export(plan, producer)
tabbyapi-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/transformer.py", line 140, in tp_export
tabbyapi-1  |     **{name: _export(getattr(self, name, None)) for name in (
tabbyapi-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/transformer.py", line 132, in _export
tabbyapi-1  |     return child.tp_export(plan, producer) if child is not None else None
tabbyapi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/attn.py", line 639, in tp_export
tabbyapi-1  |     **{name: _export(getattr(self, name, None)) for name in (
tabbyapi-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/attn.py", line 623, in _export
tabbyapi-1  |     return child.tp_export(plan, producer) if child is not None else None
tabbyapi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/linear.py", line 405, in tp_export
tabbyapi-1  |     "inner": self.inner.tp_export(plan, producer),
tabbyapi-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/modules/quant/exl3.py", line 206, in tp_export
tabbyapi-1  |     "mcg": producer.send(self.mcg_tensor),
tabbyapi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  |   File "/opt/venv/lib/python3.12/site-packages/exllamav3/model/model_tp_shared.py", line 72, in send
tabbyapi-1  |     src = t_cpu.view(torch.uint8).numpy().view(np.uint8).ravel()
tabbyapi-1  |           ^^^^^^^^^^^^^^^^^^^^^^^
tabbyapi-1  | RuntimeError: self.dim() cannot be 0 to view Int as Byte (different element sizes)

[frenzybiscuit]

Note that this model was quanted on 0.0.9

https://huggingface.co/ReadyArt/Sophosympatheia_Strawberrylemonade-L3-70B-v1.2-EXL3

[frenzybiscuit]

This isnt a docker problem! Hurray.

[figuernd]

Also experienced this after building the latest tabbyAPI; shm_size: '16gb' in docker-compose.yml fixed.