Don't apply the same patch multiple times #592

Job	Run time
ci-data	3s
Compile native gem for aarch64-linux	5m 58s
Compile native gem for aarch64-linux-musl	5m 56s
Compile native gem for arm64-darwin	5m 44s
Compile native gem for x64-mingw-ucrt	5m 41s
Compile native gem for x86_64-darwin	5m 40s
Compile native gem for x86_64-linux	5m 32s
Compile native gem for x86_64-linux-musl	5m 42s
	40m 16s

Provide feedback