MDTest.messageDigestElaborateTest() crashes on Ubuntu Pro VMs

[pushkarnk]

MDTest.messageDigestElaborateTest() has never crashed in CI, or during local development.

While testing on Ubuntu Pro with the openssl that comes with fip-updates I started observing a crash (amd64, arm64 and s390x). Here is the stack trace:

J 1036  com.canonical.openssl.md.OpenSSLMD.doUpdate0([B)V (0 bytes) @ 0x00007fe3a7a98374 [0x00007fe3a7a98320+0x0000000000000054]
J 1051 c1 com.canonical.openssl.md.OpenSSLMD.engineUpdate([B)V (72 bytes) @ 0x00007fe3a01416cc [0x00007fe3a0141400+0x00000000000002cc]
J 1047 c1 com.canonical.openssl.md.OpenSSLMD.engineUpdate(B)V (12 bytes) @ 0x00007fe3a014105c [0x00007fe3a0140f40+0x000000000000011c]
J 1053% c1 MDTest.messageDigestElaborateTest()V (293 bytes) @ 0x00007fe3a01444dc [0x00007fe3a0142460+0x000000000000207c]

This always happens after the MessageDigest.reset() call. But there is some inconsistency, I see the test failing once in 3-4 runs.

[pushkarnk]

I did some debugging and here are a few observations:

1. Increasing the Xmx value to 1G avoids the crash
2. Disabling JIT compilation (-Xint) avoids the crash
3. The crash often comes up in the loop that calls update on bytes, after reset().

This indicates a complex interplay between reset(), compiled code and resource cleanup following garbage collection.

[pushkarnk]

Closing via #104