Can you run ExifToolHelpers in multiple threads?

[kwilinsi]

I have a script that processes thousands of images with multiple worker threads. Each worker does a few things, including extract metadata with PyExifTool.

It's convenient for each worker/thread to create its own exiftool.ExifToolHelper instance. Is this safe?

It seems to work most of the time, but every once in a while I think the program is hanging on ExifToolHelper.run() (it's inconsistent and difficult to test exactly). I'm suspicious that multithreading may be the problem.

If this isn't supported, then I'll need to share one ExifToolHelper between threads (not processes). Is it thread safe, or is a lock necessary?

[kwilinsi]

I've found (at least one reason) why it hangs on Linux, along with a bad patch.

ExifTool.run() contains this line:

kwargs["preexec_fn"] = _set_pdeathsig(signal.SIGTERM)

This only executes on Linux platforms and is apparently intended to ensure that "the exiftool childprocess is stopped if this process dies" (source).

The preexec_fn parameter of subprocess.Popen has a warning in the docs (and here) that it is not safe "in the presence of threads." Indeed, commenting out the line above fixes my program: it no longer hangs.

The intended behavior here (killing the exiftool child process if my process dies) seems useful, but I can't afford reducing my program to a single thread. I don't know if preexec_fn can be replaced with something thread-safe, but perhaps this will help someone else.