Replicator annotators giving empty images on random frames when using multiple GPUs #507
Description
Description
Issue description
When using multiple GPUs (2 on my case), and a minimalistic scene (8 cameras + ground + light + BasicWriter), some of the resulting frames of some cameras are empty (full transparent image). However, RGB images for the same frame look OK.
RGB frame (all the same, static scene, dark grey ground is labelled):
Resulting semantic segmentation frames:
Screencast.from.2026-02-26.10-38-17.mp4
If single-GPU usage is forced in the docker compose file, the problem dissapears.
Other symptoms
On our replicator use case, we also found other issues realted such as some assets not being segmented even though they are labelled, or only parts of them being segmented.
Examples (doing same run multiple times):
The human actor is not fully segmented (only upper clothes) and the sky is not segmented
The human actor is not fully segmented (hand/arm missing) but the sky is segmented
Also, in our use case, we also use the PoseWriter and for frames that appear empty on the segmentation images, the PoseWriter output is empty too. This points to an annotator/rendering issue.
Reproducibility
The problem is non-deterministic but has a big occurrence rate. A full example + environment is provided that can be used to reproduce the issue. Tests have been done doing sets of 5 runs and having a ~50% of reproducibility.
The environment contains:
- README.md: file with instructions on how to run and validate the issue
- docker-compose.yml: file used to run a container with the Isaac Sim 5.1.0 official image (here we can control if using 1 or multiple gpus)
- reproduce_issue.py: a minimalistic Replicator pipeline used to reproduce the issue
- simple_validation.py: a script to validate the results and detect empty frames quickly
Some interesting facts
These are some interesting tests that could be useful hints.
With the example provided, only even cameras seem to fail. But if we swap the GPU visibility using CUDA_VISIBLE_DEVICES=1,0 then only odd cameras seem to fail.
Another interesting test is that if we use CUDA_LAUNCH_BLOCKING=1, the problem seems to focus only on Camera 4.
Isaac Sim version
5.1.0
Operating System (OS)
Ubuntu 24.04.2
GPU Name
2x NVIDIA RTX 6000 Ada Generation (47 GiB each, sm_89, mempool enabled)
GPU Driver and CUDA versions
Driver 580.126.09, CUDA 12.8 (Toolkit) 13.0 (Driver)
Logs
Additional information
Here is a zip file containing the files needed to reproduce the issue: