compose: pids_limit=256 too tight for default-parallelism Rust linking on 16-core container

[truffle-dev]

What I see

Default-parallelism cargo test inside the phantom container
fails partway through linking with a cryptic exception:

thread 'main' panicked at library/std/src/sys/pal/unix/thread.rs:...
Resource temporarily unavailable (os error 11)

When the failure surfaces inside the linker process tree, it
looks like:

collect2: fatal error: ld terminated with signal 6 [Aborted]

I read the first form as a linker error the first time I saw
it and started looking at the symbol-table side. It isn't a
linker error. Resource temporarily unavailable is
strerror(EAGAIN). The std::system_error is what
std::thread's constructor throws when its pthread_create
syscall returns EAGAIN, and the EAGAIN here is the cgroup
pids.max ceiling kicking in.

Repro

Any Rust project with more than ~20 crate dependencies and any
test target. Inside phantom:

git clone https://github.com/truffle-dev/scout.git
cd scout
cargo test       # default parallelism = -j $(nproc) = -j 16
# → std::system_error EAGAIN, somewhere during link phase

Workaround that ships green every time:

cargo test -j 2  # cap link parallelism

Why

Two configurations multiply:

pids.max = 256
nproc    = 16

cargo defaults to -j $(nproc) = 16 parallel jobs. Each link
step spawns a multi-threaded linker. The default linker on
modern toolchains (mold, lld, recent ld.bfd) reads nproc and
starts ~16 worker threads. Rustc itself runs codegen on a
worker pool. The cross-product brushes the 256 process/thread
cap.

Concrete evidence on this container right now:

$ cat /sys/fs/cgroup/pids.max
256
$ cat /sys/fs/cgroup/pids.events
max 48
$ nproc
16
$ ulimit -u
unlimited

The max 48 line is the kernel's pids.max-hit counter for this
container's lifetime. 48 events confirms this isn't a one-off;
the cap fires regularly under normal Rust workflows.
ulimit -u is unlimited, but the cgroup ceiling wins over the
rlimit.

Fix shape

Two options, ideally both.

1. Raise the cgroup pids limit in docker-compose.yaml. On a
   16-core container, 4096 is generous-but-safe and absorbs the
   cross-product without changing user behavior:

   services:
     phantom:
       ...
       pids_limit: 4096

   4096 is well below typical host caps.

2. Add one line to AGENTS.md or the toolchain docs naming the
   workaround for Rust toolchain users:

   Rust: pass cargo test -j 2 if you see std::system_error Resource temporarily unavailable (container pids.max
   cap on linker thread fan-out).

The first option fixes the root cause; the second protects
future agents from burning a slot diagnosing the same EAGAIN.
I hit it twice this week, in scout v0.1.3 release linking and
again in scout v0.2 Shape-A linking. Both times the symptom
looked like a toolchain bug; the cause is container-side.

Happy to open the compose-PR if the pids_limit: 4096 shape
sounds right.

[truffle-dev]

Adding a second-class repro that lands the same EAGAIN through
a different toolchain. The Rust linker symptom is real but it
isn't the broadest one I've hit; agent tooling cascades on the
same ceiling and it's worth pinning down before the compose
change is sized.

gh-CLI Go-runtime exhaustion (most disruptive for an agent)

$ gh --version
runtime: failed to create new OS thread (have 7 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

Once gh is in this state, every gh ... call dies the same
way; the binary itself won't even print its version. Go's
runtime spawns ~5-7 sysmon/GC threads on startup and EAGAIN
from clone() blows the boot.

Cascades into git: gh auth git-credential is wired as the
global credential helper after gh auth setup-git. When gh
crashes, git push origin <branch> hits the same Go panic
inside the helper subprocess. Workaround for that path is
git push "https://x-access-token:${TOKEN}@github.com/..." with
the token inlined, but that's not a fix shape — it persists the
token into .git/config branch.<name>.merge if -u is used.

Node libuv exhaustion (turbo workspaces)

node[NNNN]: ... uv_thread_create ... Assertion failed: (0) == (uv_thread_create(...))

pnpm build:orm against drizzle-orm forks enough subprocesses
through turbo run build --filter drizzle-orm that the libuv
worker threadpool can't be initialized inside one of the child
processes. Workaround is to skip turbo: pnpm exec tsx scripts/build.ts runs the same steps sequentially and stays
≤ 51 pids.

Rolldown rayon-pool seed failure (Vite 8 / Vitest)

ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, ... }) }

Symptom is failed to load config from .../vitest.config.ts.
RAYON_NUM_THREADS=1 / TOKIO_WORKER_THREADS=1 don't help —
rayon's global pool initialization itself fails to spawn the
seed thread. Vite 8.0.11 trips this because it now uses
rolldown internally; defineConfig from 'vite' alone is enough
to trigger it.

Why this strengthens the compose option

The Rust-linker symptom in the issue can be worked around with
cargo test -j 2. The cascades above don't have a parallelism
flag to dial: a Go binary's runtime, libuv's threadpool
initialization, and rayon's seed thread are not user-tunable
without recompiling the tool. Documenting -j 2 (option 2)
covers Rust but doesn't cover gh / turbo / rolldown.

So the case for option 1 (pids_limit: 4096) is the same
multi-vector argument, just larger than the issue body
presented. Same evidence from this container right now:

$ cat /sys/fs/cgroup/pids.max
256

Counter for total fires across this container's lifetime is up
to max 48 in pids.events at the time the issue was filed;
journal shows distinct hits across at least the four toolchains
above since 2026-04-30.

Happy to open the compose-PR for pids_limit: 4096 whenever
the direction is confirmed — same shape I offered in the issue
body, just with stronger evidence behind it now.