Issues with multi GPU rendering

[damacaa]

Hi,

I'm trying to render the same scene from two different views, in two different windows, each in a different monitor connected to a different GPU. I have set up a computer with two identical RTX5070 because I'm exploring the performance possibilities of running multiple screens with multiple GPUs.

So far I have found little success. I tried the multi gpu example, but doesn't seem to work properly on my machine. If I launch it on any of the two screens alone it works fine, but launching the program in both screens with parameters -s 0 -s 1, both windows are created, but the first one is automatically closed before it can render anything, leaving the second window working correctly.

I have also tried to implement this on my own.

m_viewer = vsg::Viewer::create();
m_options = vsg::Options::create();

// Can't create multiple devices for the same physical device (GPU)
// When a device is created, it's stored to use when creating new windows
std::map<int, vsg::ref_ptr<vsg::Device>> devices;

// Inside a loop...
int selectedGPU; // Read from parametes

if (devices.find(selectedGPU) != devices.end())
{
    // Found
    traits->device = devices[selectedGPU];

    // Create window
    auto window = vsg::Window::create(traits);
    m_windows.push_back(window);
    m_viewer->addWindow(window);

    // Create instance, surface and device
    auto instance = window->getOrCreateInstance();
    auto surface = window->getOrCreateSurface();
    auto device = window->getOrCreateDevice();
}
else
{
    // Create window
    auto window = vsg::Window::create(traits);
    m_windows.push_back(window);
    m_viewer->addWindow(window);

    // Create instance and surface
    auto instance = window->getOrCreateInstance();
    auto surface = window->getOrCreateSurface();

    // Find GPU
    auto physicalDevices = instance->getPhysicalDevices();
    vsg::ref_ptr<vsg::PhysicalDevice> selectedPhysicalDevice;
    selectedPhysicalDevice = physicalDevices[selectedGPU];

    // MANUALLY ASSIGN THE PHYSICAL DEVICE TO THE WINDOW
    if (selectedPhysicalDevice) {
        window->setPhysicalDevice(selectedPhysicalDevice);
        std::cout << "Selected: " << selectedPhysicalDevice->getProperties().deviceName << std::endl;
    }

    // When you call getOrCreateDevice() or when the viewer compiles, 
    // VSG will now use your 'selectedPhysicalDevice' and automatically 
    // handle all the tricky bits (extensions, queue families, etc.)
    auto device = window->getOrCreateDevice();
    devices[selectedGPU] = device;
}

// ...

In this case, creating 2 windows with different GPUs results in a crash in Swapchain.cpp, line 221:

VkResult result = vkCreateSwapchainKHR(*device, &createInfo, _device->getAllocationCallbacks(), &swapchain);

Unhandled exception at 0x00007FFBAD0D4116 (nvoglv64.dll) in GLTFViewerVSG.exe: An invalid parameter was passed to a function that considers invalid parameters fatal.

However, creating 2 windows with the same selectedGPU works perfectly fine.

Finally, in this last sceneario where I create multiple windows with the same GPU, I can drag one of them to a different screen connected to the other GPU and I can immediatly see that this second GPU starts being used, but its memory is almost empty. Am I right to assume this is because the first GPU is doing all the rendering and the second GPU is simply copying the resulting frames?

Thanks in advance!

Edit: I have compiled VSG with VSG_MAX_DEVICES set to 3 and I'm using the master branch.

[robertosfield]

I haven't worked with several GPU for a few years so there is chance there has been a regression. However, getting a crash in vkCreateSwapchainKHR call is something that is so early in the setup that it's less likely that there is some multi-buffering/threading issue at play, or at least so far.

Could you run the ''vsgdeviceselection --list'' example, this is what I see with my system with AMD8700G+Geforce2060.

$ vsgdeviceselection --list
vkEnumerateInstanceVersion() 4211023
VK_API_VERSION = 1.4.335.0

physicalDevices.size() = 2
    matched ref_ptr<vsg::PhysicalDevice>(vsg::PhysicalDevice 0x75d3467c7f48) NVIDIA GeForce RTX 2060, deviceType = 2, apiVersion = 1.4.312, driverVersion = 68.504.576
    QueueFamilyProperties 5
        VkQueueFamilyProperties[0] queueFlags = GRAPHICS | COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 16, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[1] queueFlags = TRANSFER | SPARSE_BINDING, queueCount = 2, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[2] queueFlags = COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 8, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[3] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_DECODE, queueCount = 1, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[4] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_ENCODE, queueCount = 1, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}

    matched ref_ptr<vsg::PhysicalDevice>(vsg::PhysicalDevice 0x75d3467c83c0) llvmpipe (LLVM 20.1.2, 256 bits), deviceType = 4, apiVersion = 1.4.318, driverVersion = 25.2.8
    QueueFamilyProperties 1
        VkQueueFamilyProperties[0] queueFlags = GRAPHICS | COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 1, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}

[damacaa]

vkEnumerateInstanceVersion() 4211001
VK_API_VERSION = 1.4.313.0

physicalDevices.size() = 3
    matched ref_ptr<vsg::PhysicalDevice>(vsg::PhysicalDevice 000001D63D125AF8) NVIDIA GeForce RTX 5070 Ti, deviceType = 2, apiVersion = 1.4.312, driverVersion = 69.228.0
    QueueFamilyProperties 6
        VkQueueFamilyProperties[0] queueFlags = GRAPHICS | COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 16, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[1] queueFlags = TRANSFER | SPARSE_BINDING, queueCount = 2, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[2] queueFlags = COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 8, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[3] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_DECODE, queueCount = 1, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[4] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_ENCODE, queueCount = 2, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[5] queueFlags = TRANSFER | SPARSE_BINDING | OPTICAL_FLOW, queueCount = 1, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}

    matched ref_ptr<vsg::PhysicalDevice>(vsg::PhysicalDevice 000001D63D125F78) NVIDIA GeForce RTX 5070 Ti, deviceType = 2, apiVersion = 1.4.312, driverVersion = 69.228.0
    QueueFamilyProperties 6
        VkQueueFamilyProperties[0] queueFlags = GRAPHICS | COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 16, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[1] queueFlags = TRANSFER | SPARSE_BINDING, queueCount = 2, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[2] queueFlags = COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 8, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[3] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_DECODE, queueCount = 1, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[4] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_ENCODE, queueCount = 2, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[5] queueFlags = TRANSFER | SPARSE_BINDING | OPTICAL_FLOW, queueCount = 1, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}

    matched ref_ptr<vsg::PhysicalDevice>(vsg::PhysicalDevice 000001D63D1263F8) NVIDIA GeForce GT 1030, deviceType = 2, apiVersion = 1.4.312, driverVersion = 69.228.0
    QueueFamilyProperties 4
        VkQueueFamilyProperties[0] queueFlags = GRAPHICS | COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 16, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[1] queueFlags = TRANSFER | SPARSE_BINDING, queueCount = 2, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[2] queueFlags = COMPUTE | TRANSFER | SPARSE_BINDING, queueCount = 8, timestampValidBits = 64, minImageTransferGranularity = {1, 1, 1}
        VkQueueFamilyProperties[3] queueFlags = TRANSFER | SPARSE_BINDING | VIDEO_DECODE, queueCount = 1, timestampValidBits = 32, minImageTransferGranularity = {1, 1, 1}

I didn't mention before but, I also have a third GPU because I need more video outputs (10 screens in total). But right now I'm only trying to make it work with 2 gpus and 2 screens.

[robertosfield]

Cool, vsgdeviceselection was able to spot all your devices even the ones you didn't mention :-)

Could you try running vsgdeviceselection by selecting the different devices i.e.

vsgdeviceselection models/teapot.vsgt --select 0
vsgdeviceselection models/teapot.vsgt --select 1
vsgdeviceselection models/teapot.vsgt --select 2

What happens when you run the vsgmultiplegpu example?

vsgmultigpu models/lz.vsgt

This was written in the early days of the VSG project when I was testing multiple GPU usage under Kubuntu. Looking at the code it uses screen number which might well limit it to X11.

Multi-GPU hasn't been something that the community have asked about so don't know the status on Windows etc, and even under Linux it's not something I've tested for a few years as the community and clients have all been pushing other aspects of the VSG.

It might be that we need to refine the vsgmultiplegpu example so it uses an approach similar to vsgdeviceselection w.r.t explicitly selecting which physical device to use.

[damacaa]

Hello Robert, sorry for taking so long to answer. I've been doing a lot of testing.

vsgmultigpu.exe models/lz.vsgt -s 0 -s 1 --no-shared

This example creates two windows, one in each screen. I had to add the --no-shared argument because otherwise I was getting two black screens. Looking at GPU stats with GPU-Z I see the same behavior I mentioned before, where GPU 0 has a much greater utilization and VRAM consumption increase, and I'm wondering if that means that this GPU is doing all the rendering and the other is simply copying the resulting frame to its window. GPU 0 is at 99% because I disabled vsync, and GPU 1 is around 30%.

Running

vsgdeviceselection models/teapot.vsgt --select 0
vsgdeviceselection models/teapot.vsgt --select 1
vsgdeviceselection models/teapot.vsgt --select 2

works as expected in all cases. Running the example and moving the window to the screen matching the selected GPU results in 60% utilization of the GPU and around 100mb of VRAM. If the window is in the wrong screen, I can see a similar pattern as in the other example: the selected GPU increases utilization a little bit, and the GPU matching the screen where the window is has around 30% utilization.

I believe vsgdeviceselection is where I learnt how to manually select a PhysicalDevice for a window, which also works on my code, as long as I only create one device and use it for all my windows.

Do you perhaps have a bigger / heavier scene available to do some more testing?

I understand multi-gpu might not be a relevant topic nowadays, so thank you very much for helping with this!

[robertosfield]

What platform are you working on?

In the vsgmultigpu case it sounds like the driver is allocating on device and then the compositor/window manager/driver is copying to the other devices screens.

The vsgmultigpu example may need updated to work more like vsgdeviceselection with querying of available PhysicalDevice and then opening windows on the appropriate screen.

[damacaa]

I'm on Windows.

Okay, I'll implement device selection in the multi gpu example and see if that works.

[damacaa]

After adding setPhysicalDevice to the multi-gpu example, it crashes again in VkResult result = vkCreateSwapchainKHR(*device, &createInfo, _device->getAllocationCallbacks(), &swapchain);

// vsgmultigpu.cpp

// EDIT: Create a container to hold the command graphs for all screens
vsg::CommandGraphs commandGraphs;

size_t numScreens = screensToUse.size();
for (size_t i = 0; i < screensToUse.size(); ++i)
{
    int screenNum = screensToUse[i];

    // create a local copy of the main WindowTraits
    auto local_windowTraits = vsg::WindowTraits::create(*windowTraits);

    // set the screenNum relative to the base one
    local_windowTraits->screenNum = screenNum;

    auto window = vsg::Window::create(local_windowTraits);
    if (!window)
    {
        std::cout << "Could not create window." << std::endl;
        return 1;
    }

    // EDIT: Set physical device
    auto instance = window->getOrCreateInstance();
    auto surface = window->getOrCreateSurface();
    auto physicalDevices = instance->getPhysicalDevices();
    auto physicalDevice = physicalDevices[i];
    window->setPhysicalDevice(physicalDevice);

    vsg::ref_ptr<vsg::Camera> camera;
    if (powerWall)
    {
        // assume a power wall layout
        auto relative_perspective = vsg::RelativeProjection::create(vsg::translate(double(numScreens - 1) - 2.0 * double(i), 0.0, 0.0), perspective);
        camera = vsg::Camera::create(relative_perspective, lookAt, vsg::ViewportState::create(window->extent2D()));
    }
    else
    {
        // assume monitors are rotated around the viewer
        double fovY = 30.0;
        double fovX = atan(tan(vsg::radians(fovY) * 0.5) * aspectRatio) * 2.0;
        double angle = fovX * (double(i) - double(numScreens - 1) / 2.0);
        auto relative_view = vsg::RelativeViewMatrix::create(vsg::rotate(angle, 0.0, 1.0, 0.0), lookAt);
        camera = vsg::Camera::create(perspective, relative_view, vsg::ViewportState::create(window->extent2D()));
    }

    auto local_scene = sharedScene ? vsg_scene : createScene(filename, options);

    // EDIT: create the command graph and add it to the list
    auto commandGraph = vsg::createCommandGraphForView(window, camera, local_scene);
    commandGraphs.push_back(commandGraph);

    viewer->addWindow(window);
}

// EDIT: add commandGraphs
viewer->assignRecordAndSubmitTaskAndPresentation(commandGraphs);