Multiple screen captures in a loop cause explorer.exe and native Windows app to lag?

[BenShapira]

Hey Rob!
My app takes a screenshot of multiple selected windows every 30 seconds in a loop.
The way the screenshots are taken is by using this service -

    public class ScreenshotService
    {
        private IDirect3DDevice device;
        private SharpDX.Direct3D11.Device d3dDevice;

        public ScreenshotService() 
        {
            device = Direct3D11Helper.CreateDevice();
            d3dDevice = Direct3D11Helper.CreateSharpDXDevice(device);
        }

        public async Task<Image> Screenshot(GraphicsCaptureItem item)
        {
            // create add texture2D 2D accessible by the CPU
            var desc = new Texture2DDescription()
            {
                Width = item.Size.Width,
                Height = item.Size.Height,
                CpuAccessFlags = CpuAccessFlags.Read,
                Usage = ResourceUsage.Staging,
                Format = Format.B8G8R8A8_UNorm,
                ArraySize = 1,
                MipLevels = 1,
                SampleDescription = new SampleDescription(1, 0),
            };

            using var cpuTexture = new Texture2D(d3dDevice, desc);

            var taskCompletionSource = new TaskCompletionSource<Direct3D11CaptureFrame>();

            using var framePool = Direct3D11CaptureFramePool.CreateFreeThreaded(
                            device,
                            DirectXPixelFormat.B8G8R8A8UIntNormalized,
                            1,
                            item.Size);

            framePool.FrameArrived += (sender, a) =>
            {
                if (!taskCompletionSource.Task.IsCompleted) taskCompletionSource.SetResult(sender.TryGetNextFrame());
            };

            using var session = framePool.CreateCaptureSession(item);

            session.StartCapture();

            using var frame = await taskCompletionSource.Task;

            using (var bitmap = Direct3D11Helper.CreateSharpDXTexture2D(frame.Surface))
            {
                // add this to copy the DirectX resource into the CPU-readable texture2D
                d3dDevice.ImmediateContext.CopyResource(bitmap, cpuTexture);
            }

            // get IDirect3DSurface from texture
            using var surf = Direct3D11Helper.CreateDirect3DSurfaceFromSharpDXTexture(cpuTexture);

            // build a WinRT's SoftwareBitmap from this surface/texture
            using var softwareBitmap = await SoftwareBitmap.CreateCopyFromSurfaceAsync(surf);

            using var InMemoryStream = new InMemoryRandomAccessStream();
            BitmapEncoder encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, InMemoryStream);

            encoder.SetSoftwareBitmap(softwareBitmap);
            await encoder.FlushAsync();            
            using var stream = InMemoryStream.AsStream();

            return Image.Load(stream);
        }

        public void Dispose()
        {
            device.Dispose();
            d3dDevice.Dispose();
        }
    }

The service is initialized once, and the Screenshot(item) method is the one that's being called in a loop.

I and a few others using my app (on Windows 10 and 11) reported odd OS lags after letting the app run for a while.
Such as -

1. Delays in opening the Windows Start menu (can take 30 seconds and even more).
2. Delays and lags in using native Windows apps such as a snipping tool. (doesn't open, or doesn't work right).
3. One user also reported that after closing the app, he still sees the screen capture yellow border mark around one of his windows, how could that even be possible?

Everything else works just fine, it's only the OS-related apps that seem to go nuts.

Looking at the app CPU/RAM consumption after a few days of running seems to be reasonable and steady -
1% CPU
80-100MB RAM

By the way, the issues above persist even after closing the app... only fixed after a restart.

Thanks, Ben

[robmikh]

The GraphicsCaptureItems are backed by shell objects, it's conceivable that there's an issue that has down stream effects on explorer. But I haven't seen this reported.

It would be interesting to know whether keeping the GraphicsCaptureItems around instead of creating new ones mitigates the issue at all.

I'll see if I can repro when I get some time.

[BenShapira]

I'll try to implement your suggestion, creating a single instance of GraphicsCaptureItems per window and keeping those in memory instead of recreating an item on every iteration.

Kind of hard to debug as this issue might take hours to reproduce, can't pinpoint it yet...
Will try reducing the interval time to speed-up the process guessing that's what causing the issue.
If you get the chance to repro and confirm I'd appreciate it if you can let me know.

Thanks

[BenShapira]

@robmikh
Ok, I'm almost certain what you said is indeed the issue.
Did a loop that's only doing -

                        GraphicsCaptureItem item = CaptureHelper.CreateItemForWindow(client.WindowHandler);

Every 100ms to speed up the process.

After a few minutes, my Windows Start menu is completely bugged, sometimes works, and sometimes takes 15 seconds to open.
The calculator app also.

Closing my App doesn't fix the issue as I mentioned before, a restart of the PC is required.
Maybe restarting explorer.exe would do the trick? I'll test .

edit -
Restarting explorer.exe also works.

[BenShapira]

@robmikh
Just an update, I've tested it during the night with 5 instances of GraphicsCaptureItem without recreating them. (using the screenshot service I shared above)

It's not as bad but the issue still occurs.
Start Menu/Calculator lags, etc.

So basically I can't see a workaround that lets me use this API without telling my users they need to restart their explorer.exe every 24h ><"

Please let me know if you can think of a hotfix/workaround that I might be able to upload until a fix is provided.
Thanks

[robmikh]

I think I'm able to repro, but just to be sure can you try running this code and see if it reproduces your issue after a few minutes? https://github.com/robmikh/manyitemstest

You'll need to use CTRL+C to exit it.

[BenShapira]

@robmikh
Ya, ran it for a few minutes and my explorer died completely.

[BenShapira]

Does this also explain why the last test I did (5 instances without recreating) also caused explorer.exe to lag after a while?

[robmikh]

Not sure yet, I'll have to debug it on Monday.

[BenShapira]

Hey Rob,
Can I ask for an update and if you can provide an ETA by any chance?

So that I can plan ahead and understand if I can wait for a fix or if I have to develop a fix with the old capture approach using user32.dll as my app is currently live and customers Explorer is dying lol

[robmikh]

Sorry but I have no ETA at this time. You may want to roll back if the issue is pressing.

[BenShapira]

Got it, nothing to roll back to, the app was built around this API.
Would appreciate an update once there is one so I can forward it to my customers, thanks.

[theonewolf]

@BenShapira what old approach would you have used? PrintWindow or BitBlt?

old capture approach using user32.dll

[BenShapira]

@BenShapira what old approach would you have used? PrintWindow or BitBlt?

old capture approach using user32.dll

Repeated PrintWindow with the framerate you want, works just fine for me.

This repo API would have been a better implementation for what I was trying to achieve. That being said, as mentioned above, using it causes explorer.exe to basically die after a short while, so it's just not usable.
Hopefully, it will get fixed eventually.

[robmikh]

Sorry for the delay on updating this issue, after fixing this I was heads down on other tasks. This was fixed as part of 24H2, and should be patched sometime early next year for 23H2. Once I get more clarity on the 23H2 timeline I can share that here.

[theonewolf]

@robmikh thanks for being a beast! I appreciate it. It will take time for enterprise customers to upgrade, but happy to hear this is coming.