Skip to content

RayDP ray_dist.jar fix #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
avigyabb wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

RayDP ray_dist.jar fix #39

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f41478c
working
avigyabb Jan 21, 2026
6c474fb
unnecessary code
avigyabb Jan 21, 2026
3b39baf
clean
avigyabb Jan 22, 2026
82f8b75
clean
avigyabb Jan 26, 2026
ffdd92d
clean
avigyabb Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions spark_on_ray/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Anyscale Container-Compatible Dockerfile
FROM anyscale/ray:2.50.0-slim-py312-cu128
FROM anyscale/ray:2.53.0-slim-py312-cu128

# Environment variables
ENV ANYSCALE_DISABLE_OPTIMIZED_RAY=1
# ENV ANYSCALE_DISABLE_OPTIMIZED_RAY=1
ENV DEBIAN_FRONTEND=noninteractive
ENV HOME=/home/ray
ENV PATH=/home/ray/anaconda3/bin:$PATH
Expand Down Expand Up @@ -99,6 +99,17 @@ RUN /home/ray/anaconda3/bin/pip install --no-cache-dir py4j
# Install raydp
RUN /home/ray/anaconda3/bin/pip install --no-cache-dir --pre raydp

# Download ray_dist.jar from PyPI and backup for restoration after pre-start
# The Anyscale pre-start script replaces Ray at container startup, removing the JAR.
RUN cd /tmp && \
/home/ray/anaconda3/bin/pip download "ray==2.53.0" --no-deps -d . && \
sudo mkdir -p /opt/ray-jars-backup && \
sudo unzip -o -j ray-*.whl "ray/jars/ray_dist.jar" -d /opt/ray-jars-backup/ && \
rm -f ray-*.whl

# Patch pre-start script to restore ray_dist.jar after Ray replacement
RUN [ -f /opt/anyscale/ray-prestart ] && printf '\n# RAYDP FIX: Restore ray_dist.jar\nif [[ -f /opt/ray-jars-backup/ray_dist.jar ]]; then\n "${SUDO[@]}" mkdir -p "${ANYSCALE_RAY_SITE_PKG_DIR}/ray/jars"\n "${SUDO[@]}" cp /opt/ray-jars-backup/ray_dist.jar "${ANYSCALE_RAY_SITE_PKG_DIR}/ray/jars/"\nfi\n' | sudo tee -a /opt/anyscale/ray-prestart > /dev/null || true

# Install additional Python packages for Spark/Hadoop integration
RUN /home/ray/anaconda3/bin/pip install --no-cache-dir emoji pyarrow pandas numpy findspark

Expand Down