Crash on RoUninitialize or FreeLibrary of graphicscapture.dll unless first sleeping for a while

[BtbN]

Hi!

We've recently added Windows.Graphics.Capture support to FFmpeg, but are facing some difficulty with the correct cleanup procedure.

Specifically, since we are a library that can't expect anything from our calling process, and also don't want to affect it adversely, we have confined all interaction with WinRT/WGC into its own thread.
That thread initially calls RoInitialize in multithreaded mode, sets up a dispatch queue for that thread, and after setting up the capture chain, just sits there and pumps messages until a custom shutdown message comes in, which makes it Close() the frame pool and capture session.

So far this all works. But when the thread is about to exit, and has cleaned up everything else, it eventually will call RoUninitialize, and at that point it just crashes most of the time (sometimes it gets through it, especially if a debugger is attached).

With some experimentation, we also found out that just adding a plain old sleep() for 100 ms or so right before RoUninit fixes this. But obviously it isn't a pretty thing to do, and also seems random and could just fail as well.
Then we figured out that just LoadLibrary()'ing graphicscapture.dll results in RoUninit never crashing. But instead now we get a crash when freeing the handle, but much much less frequent. Most of the time it exits cleanly. Seemingly because enough time passes until we actually free the library.
From the looks of it, some background thread created internally is still busy spinning down and not properly joined?
Can't think of much else that'd exhibit this behaviour that gets "fixed" with sleeping.

Here's one of the crash logs I managed to capture:

#0  0x00007fffdaf766ca in RaiseException () from C:\WINDOWS\System32\KernelBase.dll
#1  0x00007fffdac9fe9f in msvcp_win!?__ExceptionPtrRethrow@@YAXPEBX@Z () from C:\WINDOWS\System32\msvcp_win.dll
#2  0x00007fff88b46d4c in GraphicsCapture!DllGetActivationFactory () from C:\Windows\System32\GraphicsCapture.dll
#3  0x000000000a4ba518 in ?? ()
#4  0x000000000a4ba4e0 in ?? ()
#5  0x0000000000000000 in ?? ()

While this was going on, the ffmpeg capture thread was doing this:

Thread 7 (Thread 23124.0x7ce8 "fc0"):
#0  0x00007fffdd963774 in ntdll!ZwUnmapViewOfSection () from C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x00007fffdd824a86 in ntdll!RtlAvlInsertNodeEx () from C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x00007fffdd848d0a in ntdll!RtlQueryInformationActivationContext () from C:\WINDOWS\SYSTEM32\ntdll.dll
#3  0x00007fffdd84776f in ntdll!LdrUnloadDll () from C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x00007fffdaecde19 in KERNELBASE!FreeLibrary () from C:\WINDOWS\System32\KernelBase.dll
#5  0x00007fffdd1f03be in combase!CoFreeUnusedLibrariesEx () from C:\WINDOWS\System32\combase.dll
#6  0x00007fffdd1ef945 in combase!CoInitializeEx () from C:\WINDOWS\System32\combase.dll
#7  0x00007fffdd1f0b54 in combase!CoFreeUnusedLibrariesEx () from C:\WINDOWS\System32\combase.dll
#8  0x00007fffdd19de38 in combase!RoOriginateErrorW () from C:\WINDOWS\System32\combase.dll
#9  0x00007fffdd19e236 in combase!RoOriginateErrorW () from C:\WINDOWS\System32\combase.dll
#10 0x00007fffdd19e36d in combase!RoOriginateErrorW () from C:\WINDOWS\System32\combase.dll
#11 0x00007fffdd1ed707 in combase!CoUninitialize () from C:\WINDOWS\System32\combase.dll
#12 0x00007fffdd1ed559 in combase!RoUninitialize () from C:\WINDOWS\System32\combase.dll
#13 0x00007ff77cf7c6e7 in gfxcapture_uninit (avctx=<optimized out>) at libavfilter/vsrc_gfxcapture.c:390
#14 0x00007ff77b94bf76 in avfilter_free (filter=0x266d540) at libavfilter/avfilter.c:807
#15 0x00007ff77b94e624 in avfilter_graph_free (graphp=0x2c2f9b0) at libavfilter/avfiltergraph.c:128
#16 0x00007ff77b8f22bf in fg_thread_uninit (fgt=<optimized out>) at fftools/ffmpeg_filter.c:3028
#17 filter_thread (arg=0x26615c0) at fftools/ffmpeg_filter.c:3178
#18 0x00007ff77b904f97 in task_wrapper (arg=0x26661b0) at fftools/ffmpeg_sched.c:2534
#19 0x00007ff77b903acf in win32thread_worker (arg=0x26695c0) at ./compat/w32pthreads.h:79
#20 0x00007fffdc82f0ad in msvcrt!_beginthreadex () from C:\WINDOWS\System32\msvcrt.dll
#21 0x00007fffdc82f17c in msvcrt!_endthreadex () from C:\WINDOWS\System32\msvcrt.dll
#22 0x00007fffdc5ee8d7 in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\System32\kernel32.dll
#23 0x00007fffdd808d9c in ntdll!RtlUserThreadStart () from C:\WINDOWS\SYSTEM32\ntdll.dll
#24 0x0000000000000000 in ?? ()

This was without manually loading the graphicscapture.dll, so supposedly it was freeing the graphicscapture.dll.

The respective code is here:
https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vsrc_gfxcapture_winrt.cpp#L500
All the functions prefixed wgc_ are running on the thread, and it's where largely all the relevant code is confined.

[robmikh]

Thanks for reporting this!

Do you have a repro application I could build/run to debug this? Additionally, could you file a feedback item in the Feedback Hub app and attach a memory dump of the crash? That'll help me investigate.

[BtbN]

https://btbn.de/files/gfx_crashing_ffmpeg.zip here's a binary of the crashing ffmpeg with a PDB file, built with VS2022.
For a simple reproducer, I usually use this:
./ffmpeg_g.exe -filter_complex gfxcapture=monitor_idx=0:max_framerate=60 -frames 5 -f null -

For me it crashes in roughly one of 10 attempts, so I usually just spam it until it crashes to reproduce the crash.
Annoyingly, running in a debugger makes it nearly impossible to reproduce the crash, due to the slower running code avoiding the race almost entirely. It's still sometimes reproducible though.

I'll look into the memory dump via feedback hub, not quite sure yet how to capture one.

Edit: The binary above is just a plain build of FFmpeg master with this one line commented out, so it doesn't LoadLibrary the graphicscapture.dll anymore:
https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vsrc_gfxcapture_winrt.cpp#L1014

[BtbN]

Ok, managed to collect one, and submitted as https://aka.ms/AAxzvql

[kasper93]

I can add two cents here. This may be another instance of crash, so please first look at @BtbN case.

I can consistently catch crashing thread [1], which seems to call into winrt::impl::signal_awaiter::callback (looking at debug symbols) in GraphicsCapture.dll. The issue is RoUninitialize which basically calls CoUninitialize is killing all COM on this thread, including unloading of GraphicsCapture.dll library, which were previously loaded by querying activation factory.

The issue is that, to the best of our knowledge, we cleared all our COM objects, but there seems to be no way to tear down the internal GraphicsCapture things.

For more direct reproduce you should remove this line https://github.com/FFmpeg/FFmpeg/blob/1ce3f9fdabf0c769e86b585d9da197e869e1a44e/libavfilter/vsrc_gfxcapture_winrt.cpp#L1014 which basically manually loads GraphicsCapture.dll, so it is loaded "longer". This greatly reduces crashes occurrence, but it's only a hack to delay unloading of the library, in hope that whatever need to end internally ends...

[1] Crash in callback after unloading dll.

[0x0]   <Unloaded_GraphicsCapture.dll>+0x21420   0x37cc0ff7a8   0x7ffe9c6bddbd   
[0x1]   ntdll!TppExecuteWaitCallback+0x4cd   0x37cc0ff7b0   0x7ffe9c6d2a10   
[0x2]   ntdll!TppWorkerThread+0x820   0x37cc0ff8d0   0x7ffe9a71e8d7   
[0x3]   KERNEL32!BaseThreadInitThunk+0x17   0x37cc0ffc30   0x7ffe9c67c34c   
[0x4]   ntdll!RtlUserThreadStart+0x2c   0x37cc0ffc60   0x0   

[robmikh]

Thanks for the dump and the extra info, everyone!

I spent some time trying to repro this on two different machines and haven't had any luck. However, I think I have some idea of what's going on and how to fix it. I've set up a synthetic repro and the fix seems to do the job. I'll get that process started internally.

The one wrinkle here is that my fix only addresses the frame pool's internal thread, and likely what @BtbN is seeing. What @kasper93 is seeing is related to the GraphicsCaptureItem. I'm going to need a little more time to think about the proper fix for that one.

[kasper93]

Thank you for looking into it, much appreciated.

To reproduce, I had better luck with ASAN. It catches unknown address reads even late in process lifetime and is more reproducible than attaching debugger.

Application Verifier works too to fine memory leaks, but in latest version it wasn't catching any IIRC.