Skip to content

chore(deps): update dependency pyspark to v4 #13387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
renovate-bot wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore(deps): update dependency pyspark to v4 #13387

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pubsublite/spark-connector/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
pyspark[sql]==3.5.5
pyspark[sql]==4.0.0

@gemini-code-assist gemini-code-assist Bot May 23, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Upgrading to PySpark 4.0.0 is a significant change. Apache Spark 4.0 now uses Scala 2.13 by default, whereas Spark 3.x versions (like 3.5.5) primarily used Scala 2.12.

This raises a critical concern about the compatibility of the Pub/Sub Lite Spark connector JARs used in your examples and tests (spark_streaming_test.py):

  • gs://pubsublite-spark/pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar
  • gs://spark-lib/pubsublite/pubsublite-spark-sql-streaming-LATEST-with-dependencies.jar

These JARs, especially version 1.0.0, are likely compiled against Scala 2.12. For instance, the java-pubsublite-spark connector version 1.1.0 (released Feb 2024) specifies Spark 3.3.2 and Scala 2.12.

Could you please verify the following points?

  1. Are there versions of the Pub/Sub Lite Spark connector JARs available that are compiled for Spark 4.0 and Scala 2.13?
  2. Have the paths in spark_streaming_test.py been updated to use these compatible JARs if necessary?
  3. Have all tests in spark_streaming_test.py been executed successfully with PySpark 4.0 and the (potentially new) connector JARs? Without compatible connectors, these tests are very likely to fail due to binary incompatibilities between Scala versions.

Additionally, PySpark 4.0 may introduce other breaking changes or behavior modifications (e.g., the default change for spark.sql.legacy.respectNullabilityInTextDatasetConversion). It's important to review the Apache Spark 4.0.0 release notes for any other changes that might affect your examples and ensure they behave as expected.