JDK26 support with lates spark 4.2-preview5

[andrey-borisov-patrianna]

---

Spark 4.2.0-preview5 fails at SparkContext init on JDK 26: ClassNotFoundException: jdk.internal.ref.Cleaner

Description

org.apache.spark.unsafe.Platform fails to initialize on JDK 26 because its
static initializer attempts to load jdk.internal.ref.Cleaner by name, which no
longer exists / is no longer accessible in JDK 26. The failure surfaces as an
ExceptionInInitializerError the first time SparkSession.builder().getOrCreate()
creates a SparkContext, making Spark completely unusable on JDK 26.

The root cause is in Platform.<clinit> (Platform.java:84 / :104), which resolves a
Cleaner class via Class.forName("jdk.internal.ref.Cleaner") and rethrows the
ClassNotFoundException wrapped in an IllegalStateException.

Affected version

• Spark: 4.2.0-preview5
• Scala: 2.13.18
• JDK: OpenJDK 26.0.1 (Homebrew build 26.0.1)
• OS: macOS (Darwin 25.5.0)

Steps to reproduce

1. Build/run with JDK 26.
2. Create a local SparkSession:

   SparkSession.builder()
       .master("local[*]")
       .getOrCreate();

Actual behavior

java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.(ByteArrayMethods.java:47)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:261)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:251)
at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:271)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.memory.MemoryManager.(MemoryManager.scala:271)
at org.apache.spark.memory.UnifiedMemoryManager.(UnifiedMemoryManager.scala:67)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:456)
at org.apache.spark.SparkEnv.initializeMemoryManager(SparkEnv.scala:261)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:456)
at org.apache.spark.SparkEnv.initializeMemoryManager(SparkEnv.scala:261)
at org.apache.spark.SparkContext.(SparkContext.scala:597)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:3075)
at org.apache.spark.sql.classic.SparkSession$Builder.$anonfun$build$2(SparkSession.scala:1045)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.sql.classic.SparkSession$Builder.build(SparkSession.scala:1036)
at org.apache.spark.sql.classic.SparkSession$Builder.getOrCreate(SparkSession.scala:1065)
at org.apache.spark.sql.classic.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:929)
Caused by: java.lang.IllegalStateException: java.lang.ClassNotFoundException: jdk.internal.ref.Cleaner
at org.apache.spark.unsafe.Platform.(Platform.java:104)
... 19 more
Caused by: java.lang.ClassNotFoundException: jdk.internal.ref.Cleaner
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:502)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:478)
at java.base/java.lang.Class.forName(Class.java:468)
at org.apache.spark.unsafe.Platform.(Platform.java:84)
... 19 more

Expected behavior

Platform should initialize successfully on JDK 26 and SparkContext creation
should complete. Platform. should tolerate the absence of
jdk.internal.ref.Cleaner (e.g. fall back gracefully when the class cannot be
resolved) rather than propagating the ClassNotFoundException as an
ExceptionInInitializerError.

Notes

• Reproduces deterministically on every getOrCreate() — it is not
  timing/environment dependent.
• Likely also requires the usual --add-opens/--add-exports JVM flags for
  java.base internals on recent JDKs, but the missing class here is a hard
  removal, not merely an access restriction, so JVM flags alone do not resolve it.

---

[krishnamanchikalapudi]

Hi @andrey-borisov-patrianna, As a CodePath student, I have looked into the behavior test failures listed in this tracking issue. I am currently analyzing the code and plan to work on a resolution for this as part of my course project.

[krishnamanchikalapudi]

Hi @andrey-borisov-patrianna, opened a fix: #56952 - Platform's static initializer now catches ClassNotFoundException/NoSuchMethodException around the jdk.internal.ref.Cleaner lookup and falls back to CLEANER_CREATE_METHOD = null, the same fallback already used when --add-opens java.base/jdk.internal.ref=ALL-UNNAMED isn't granted, instead of letting the missing class abort static init.
CI is in progress on the fork; Check changes has passed so far, remaining jobs (build, Java 25 Maven build, UI tests, linters, docs) are still running.

[pan3793]

@andrey-borisov-patrianna, Spark 4.2 is supposed to be compatible with Java 17, 21, and 25. Spark has no guarantee to support non-LTS Java versions.

@krishnamanchikalapudi, masking the issue does not seem to be the right direction to fix it, we should make the cleaner work on Java 26 instead.

[andrey-borisov-patrianna]

Agreed — I think the right move is to reconsider Spark's dependency on the now-removed jdk.internal.ref.Cleaner rather than work around it, and to get this into the 4.2 line rather than defer it.

JDK 26 is a stable, GA release — not an early-access build

JDK 26 reached general availability in March 2026 as a production-ready feature release under the standard six-month cadence. This isn't a preview or EA JDK we're testing speculatively; it's a shipping release that users are already adopting, so the jdk.internal.ref.Cleaner removal is a hard break against a supported runtime, not a moving target.

Spark 4.2 is still in preview — this is the ideal window

Because 4.2.0 hasn't gone GA yet, addressing this now means JDK 26 support lands cleanly in the first stable 4.2 release rather than requiring a follow-up patch. Preview is exactly the phase where new-JDK compatibility issues are meant to be caught and resolved.

Why JVM flags won't cut it

Since jdk.internal.ref.Cleaner is a hard removal in JDK 26 (not just an access restriction), --add-opens / --add-exports can't bring it back. The static initializer in Platform.<clinit> resolves the class via Class.forName(...) and rethrows the ClassNotFoundException as an ExceptionInInitializerError, so any code path that touches Platform fails immediately on JDK 26.

Proper solution

I'd lean toward migrating off the internal API entirely — falling back to the public java.lang.ref.Cleaner (available since JDK 9), or dropping the internal-cleaner path where it's no longer needed. That keeps Spark aligned with supported JDK APIs going forward instead of chasing each internal-class removal.

Feature flag as a bridge

If a full migration is too large for the 4.2 timeline, a feature flag (or graceful degradation) that lets Platform initialize without the internal Cleaner when it can't be resolved would at least unblock JDK 26 users, with the cleaner solution landing afterward.

Since 4.2 is still in preview, this feels like the right window to decide on the direction. Happy to help test either approach against OpenJDK 26.0.1.

[pan3793]

@andrey-borisov-patrianna, teach your AI agent to read the Spark version policy and the release notes history instead of relying on common knowledge to make statements.

Spark 4.2 is in the RC phase; no more features will be accepted.

Java 26 is a stable version, but it is a non-LTS version, the current Java LTS version is 25, and the next LTS version is 29. Spark depends on many third-party dependencies, and it's unrealistic to fix all of them to be compatible with a new JDK in a short time, this generally takes years to complete even with active participation. That's why many projects choose to support Java LTS versions only.

I raised a fix #56954 to make the cleaner work on JDK 26, you are welcome to review and test this patch. If you have a different idea, please open a new PR to demonstrate your proposal.

I'd lean toward migrating off the internal API entirely — falling back to the public java.lang.ref.Cleaner (available since JDK 9), or dropping the internal-cleaner path where it's no longer needed.

[andrey-borisov-patrianna]

@pan3793 thank you for creating the patch it looks ok to me 👍 you can count on me as early adopter once this is merge and included in whatever new milestone - i will provide feedback