CryptographicException in NonPowerOfTwoKeySizeOaepRoundtrip test

[MihaZupan]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1170333
Build error leg or test failing: System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.NonPowerOfTwoKeySizeOaepRoundtrip
Pull request: #119673

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["NonPowerOfTwoKeySizeOaepRoundtrip", "CryptographicException"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.NonPowerOfTwoKeySizeOaepRoundtrip(oaepPaddingMode: OaepSHA3-512) [FAIL]
      System.Security.Cryptography.CryptographicException : Error occurred during a cryptographic operation.
      Stack Trace:
        /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.Rsa.cs(46,0): at Interop.Crypto.RsaGenerateKey(Int32 keySize)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(724,0): at System.Security.Cryptography.RSAOpenSsl.GenerateKey()
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(327,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ViaFactory(LazyThreadSafetyMode mode)
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(345,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(425,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].CreateValue()
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(505,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].get_Value()
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(712,0): at System.Security.Cryptography.RSAOpenSsl.GetKey()
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(236,0): at System.Security.Cryptography.RSAOpenSsl.TryEncrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(52,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.<>c__DisplayClass0_0.<Encrypt>b__0(Byte[] dest)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(62,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.TryWithOutputArray(Func`2 func)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(52,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.Encrypt(RSA rsa, Byte[] data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(701,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.NonPowerOfTwoKeySizeOaepRoundtrip(RSAEncryptionPadding oaepPaddingMode)
           at System.Object.InvokeStub_EncryptDecrypt.NonPowerOfTwoKeySizeOaepRoundtrip(Object , Span`1 )
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(95,0): at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1170333
Error message validated: [at Interop.Crypto.RsaGenerateKey Error occurred during a cryptographic operation]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 10/10/2025 6:00:46 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1170333
Error message validated: [NonPowerOfTwoKeySizeOaepRoundtrip CryptographicException]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 10/10/2025 6:23:48 PM UTC

Report

Build	Definition	Test	Pull Request
1346111	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125882
1343896	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125329
1343284	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125505
1340392	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125710
1340155	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125698
1340047	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125468
1338972	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124893
1338061	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125556
1337957	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125468
1335742	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125559
1335573	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125512
1334923	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125426
1333958	dotnet/runtime	System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_Array.NonPowerOfTwoKeySizeOaepRoundtrip	#123252
1331991	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125438
1329021	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124164
1328999	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125359
1327971	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124213
1324953	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124375
1319209	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125080
1318441	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125138
1318285	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#123323
1318148	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124918
1318058	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124959
1316114	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125056
1313537	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124977
1309937	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124858
1307000	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124738
1305436	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124641
1304990	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#124738
1302166	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#123890

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	9	29

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• STATUS_UNSUCCESSFUL in RsaCryptRoundtrip_OaepSHA1 #29683

[danmoseley]

Analysis (via GitHub Copilot)

The test NonPowerOfTwoKeySizeOaepRoundtrip calls RSA.Create(3072), which lazily invokes OpenSSL's EVP_PKEY_keygen via the native shim CryptoNative_RsaGenerateKey. On CI, this occasionally fails with a CryptographicException — likely due to entropy starvation or resource contention on shared build agents.

There is no retry logic anywhere in the call chain: the native shim returns NULL on any OpenSSL failure, and the managed interop immediately throws.

Possible approaches

Approach	Description	Pros	Cons
A. RetryHelper in test	Wrap test body with RetryHelper.Execute(..., retryWhen: e => e is CryptographicException)	Minimal, follows existing patterns (used in X509 revocation tests), no production code changes	Only fixes this specific test
B. Retry in native shim	Retry EVP_PKEY_keygen 2-3 times in CryptoNative_RsaGenerateKey	Fixes all callers	Changes production behavior; hard to distinguish transient from permanent failures
C. Retry in managed interop	Retry in Interop.Crypto.RsaGenerateKey	Same as B but in managed code	Same risks as B
D. [ActiveIssue] skip	Skip the test referencing this issue	Stops CI noise immediately	Loses test coverage

Option A seems like the best fit — lowest risk, follows existing repo conventions, and doesn't modify production code. If other RSA keygen tests also flake, the same pattern can be extended.