Skip to content

[CORE][MINOR] Expose Gluten build information in Spark configuration#11838

Open
jiangjiangtian wants to merge 1 commit intoapache:mainfrom
jiangjiangtian:set_build_info
Open

[CORE][MINOR] Expose Gluten build information in Spark configuration#11838
jiangjiangtian wants to merge 1 commit intoapache:mainfrom
jiangjiangtian:set_build_info

Conversation

@jiangjiangtian
Copy link
Contributor

What changes are proposed in this pull request?

Currently, Gluten build information is not injected into the Spark configuration, which causes difficulties in certain troubleshooting and tracking scenarios. For example:

  1. To find out which specific Git revision a job is running on, we have to grep the driver's syslog. Since driver logs are deleted much earlier than Spark History logs, this information becomes unavailable once the syslog is purged.
  2. We have a tracking table that stores all configs of our running jobs. Without the build metadata in the config, it's hard to tell when a job starts running on a newer version of Gluten.
    Therefore, this PR sets the Gluten build information as part of the Spark configurations. With this change, we can simply check the build revision by looking at the Spark UI:
image

@github-actions github-actions bot added the CORE works for Gluten Core label Mar 27, 2026
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant