Skip to content

fix: multiple OpenSSL 3.x memory leaks#75

Draft
Koan-Bot wants to merge 1 commit intocpan-authors:mainfrom
atoomic:koan.atoomic/fix-memleak-get-key-params
Draft

fix: multiple OpenSSL 3.x memory leaks#75
Koan-Bot wants to merge 1 commit intocpan-authors:mainfrom
atoomic:koan.atoomic/fix-memleak-get-key-params

Conversation

@Koan-Bot
Copy link
Contributor

Summary

Fixes three distinct memory leak families in OpenSSL >= 3.0.0 code paths:

  • _get_key_parameters(): EVP_PKEY_get_bn_param() allocates new BIGNUMs (unlike pre-3.x getters which return internal const pointers), but cor_bn2sv() duplicates them via BN_dup() without freeing the originals — 8 BIGNUMs leaked per call
  • verify(): XSRETURN_NO/XSRETURN_YES returned immediately, bypassing EVP_MD_free() and EVP_PKEY_CTX_free() cleanup — leaked on every verify call
  • _new_key_from_parameters(): EVP_PKEY_CTX (pctx) was never freed on any path, test_ctx from EVP_PKEY_check() was never freed, and OSSL_PARAM_BLD/OSSL_PARAM were not freed in the public-key-only branch

Root cause

The OpenSSL 3.x migration changed ownership semantics: pre-3.x getter functions return internal pointers (caller must not free), while 3.x functions like EVP_PKEY_get_bn_param() allocate new objects (caller must free). The original migration missed several of these ownership changes.

Test plan

  • All 278 tests pass (including bignum tests that exercise _get_key_parameters())
  • Compiled with OpenSSL 3.6.1 on macOS
  • CI will validate against OpenSSL 1.1.1 (bullseye), 3.0.x (bookworm), 3.4.x (trixie)

🤖 Generated with Claude Code

… construction

Three distinct memory leak fixes for OpenSSL >= 3.0.0 code paths:

1. _get_key_parameters(): EVP_PKEY_get_bn_param() allocates new BIGNUMs
   (unlike pre-3.x getters which return internal pointers). cor_bn2sv()
   duplicates them via BN_dup() but the originals were never freed,
   leaking 8 BIGNUMs on every call.

2. verify(): XSRETURN_NO/XSRETURN_YES returned immediately, bypassing
   EVP_MD_free(md) and EVP_PKEY_CTX_free(ctx) cleanup. Restructured to
   capture verify result, free resources, then switch on the result.

3. _new_key_from_parameters():
   - EVP_PKEY_CTX (pctx) was never freed on any path (success or error)
   - EVP_PKEY_CTX from EVP_PKEY_check() (test_ctx) was never freed
   - OSSL_PARAM_BLD and OSSL_PARAM were not freed in the else branch
     (public-key-only path)
   Moved pctx/params_build/params declarations to PREINIT for proper
   scope, added cleanup on both success and error paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant