You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/source/ray-contribute/development.rst
+57-60Lines changed: 57 additions & 60 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
.. _building-ray:
2
2
3
-
Building Ray from Source
3
+
Building Ray from source
4
4
=========================
5
5
6
6
To contribute to the Ray repository, follow the instructions below to build from the latest master branch.
@@ -85,12 +85,12 @@ Create a virtual environment to prevent version conflicts and to develop with an
85
85
86
86
.. _python-develop:
87
87
88
-
Building Ray (Python Only)
88
+
Building Ray (Python only)
89
89
--------------------------
90
90
91
91
.. note:: Unless otherwise stated, directory and file paths are relative to the project root directory.
92
92
93
-
RLlib, Tune, Autoscaler, and most Python files do not require you to build and compile Ray. Follow these instructions to develop Ray's Python files locally without building Ray.
93
+
RLlib, Tune, Autoscaler, and most Python files don't require you to build and compile Ray. Follow these instructions to develop Ray's Python files locally without building Ray.
94
94
95
95
1. Make sure you have a clone of Ray's git repository (see :ref:`fork-ray-repo`).
96
96
@@ -103,15 +103,15 @@ RLlib, Tune, Autoscaler, and most Python files do not require you to build and c
4. Replace Python files in the installed package with your local editable copy. We provide a simple script to help you do this: ``python python/ray/setup-dev.py``. Running the script will remove the ``ray/tune``, ``ray/rllib``, ``ray/autoscaler`` dir (among other directories) bundled with the ``ray`` pip package, and replace them with links to your local code. This way, changing files in your git clone will directly affect the behavior of your installed Ray.
106
+
4. Replace Python files in the installed package with your local editable copy. Use the script ``python python/ray/setup-dev.py`` to do this. The script removes the ``ray/tune``, ``ray/rllib``, ``ray/autoscaler`` directories (among others) bundled with the ``ray`` pip package, replacing them with links to your local code. This way, changing files in your git clone directly affects the behavior of your installed Ray.
107
107
108
108
.. code-block:: shell
109
109
110
110
# This replaces `<package path>/site-packages/ray/<package>`
111
111
# with your local `ray/python/ray/<package>`.
112
112
python python/ray/setup-dev.py
113
113
114
-
.. note:: [Advanced] You can also optionally skip creating symbolic link for directories of your choice.
114
+
.. note:: [Advanced] You can also optionally skip creating symbolic links for specific directories:
115
115
116
116
.. code-block:: shell
117
117
@@ -120,7 +120,7 @@ RLlib, Tune, Autoscaler, and most Python files do not require you to build and c
120
120
121
121
.. _python-develop-uninstall:
122
122
123
-
.. warning:: Do not run ``pip uninstall ray`` or ``pip install -U`` (for Ray or Ray wheels) if setting up your environment this way. To uninstall or upgrade, you must first ``rm -rf`` the pip-installation site (usually a directory at the ``site-packages/ray`` location), then do a pip reinstall (see the command above), and finally run the above ``setup-dev.py`` script again.
123
+
.. warning:: Don't run ``pip uninstall ray`` or ``pip install -U`` (for Ray or Ray wheels) if setting up your environment this way. To uninstall or upgrade, first ``rm -rf`` the pip-installation site (usually a directory at the ``site-packages/ray`` location), then do a pip reinstall (see the command above), and finally run the ``setup-dev.py`` script again.
124
124
125
125
.. code-block:: shell
126
126
@@ -131,18 +131,18 @@ RLlib, Tune, Autoscaler, and most Python files do not require you to build and c
131
131
132
132
.. _full-source-build:
133
133
134
-
Full Source Build
134
+
Full source build
135
135
-----------------
136
136
137
137
.. tip::
138
138
139
139
If you already followed the instructions in :ref:`python-develop` and want to switch
140
-
to the Full build, you will need to first uninstall Ray (see :ref:`uninstallation steps <python-develop-uninstall>`).
140
+
to the Full build, you need to first uninstall Ray (see :ref:`uninstallation steps <python-develop-uninstall>`).
141
141
142
142
Preparing to build Ray on Linux
143
143
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
144
144
145
-
.. tip:: If you are only editing Tune/RLlib/Autoscaler files, follow instructions for :ref:`python-develop` to avoid long build times.
145
+
.. tip:: If you're only editing Tune/RLlib/Autoscaler files, follow :ref:`python-develop` instead to avoid long build times.
146
146
147
147
To build Ray on Ubuntu, run the following commands:
148
148
@@ -159,10 +159,7 @@ To build Ray on Ubuntu, run the following commands:
159
159
nvm install 14
160
160
nvm use 14
161
161
162
-
.. note::
163
-
The `install-bazel.sh` script installs `bazelisk` for building Ray.
164
-
Note that `bazel` is installed at `$HOME/bin/bazel`; make sure it's on the executable `PATH`.
165
-
If you prefer to use `bazel`, only version `6.5.0` is currently supported.
162
+
.. note:: The ``install-bazel.sh`` script installs ``bazelisk``. Note that ``bazel`` is installed at ``$HOME/bin/bazel``; make sure it's on your ``PATH``. If you prefer to use ``bazel`` directly, only version ``6.5.0`` is currently supported.
166
163
167
164
For RHELv8 (Redhat EL 8.0-64 Minimal), run the following commands:
168
165
@@ -173,12 +170,12 @@ For RHELv8 (Redhat EL 8.0-64 Minimal), run the following commands:
173
170
174
171
In RedHat, install Bazel manually from this link: https://bazel.build/versions/6.5.0/install/redhat
175
172
176
-
Preparing to build Ray on MacOS
173
+
Preparing to build Ray on macOS
177
174
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
178
175
179
-
.. tip:: Assuming you already have Brew and Bazel installed on your mac and you also have grpc and protobuf installed on your mac consider removing those (grpc and protobuf) for smooth build through the commands ``brew uninstall grpc``, ``brew uninstall protobuf``. If you have built the source code earlier and it still fails with errors like ``No such file or directory:``, try cleaning previous builds on your host by running the commands ``brew uninstall binutils`` and ``bazel clean --expunge``.
176
+
.. tip:: If you have grpc or protobuf installed, remove them first for a smooth build: ``brew uninstall grpc``, ``brew uninstall protobuf``. If the build fails with ``No such file or directory`` errors, clean previous builds with ``brew uninstall binutils`` and ``bazel clean --expunge``.
180
177
181
-
To build Ray on MacOS, first install these dependencies:
178
+
To build Ray on macOS, first install these dependencies:
182
179
183
180
.. code-block:: bash
184
181
@@ -188,31 +185,31 @@ To build Ray on MacOS, first install these dependencies:
188
185
# Install Bazel.
189
186
ci/env/install-bazel.sh
190
187
191
-
Building Ray on Linux & MacOS (full)
188
+
Building Ray on Linux & macOS (full)
192
189
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193
190
194
-
Make sure you have a local clone of Ray's git repository (see :ref:`fork-ray-repo`). You will also need to install NodeJS_ to build the dashboard.
191
+
Make sure you have a local clone of Ray's git repository (see :ref:`fork-ray-repo`). You also need to install NodeJS_ to build the dashboard.
195
192
196
193
Enter into the project directory, for example:
197
194
198
195
.. code-block:: shell
199
196
200
197
cd ray
201
198
202
-
Now you can build the dashboard. From inside of your local Ray project directory enter into the dashboard client directory:
199
+
Build the dashboard. From inside your local Ray project directory, enter the dashboard client directory:
203
200
204
201
.. code-block:: bash
205
202
206
203
cd python/ray/dashboard/client
207
204
208
-
Then you can install the dependencies and build the dashboard:
205
+
Install the dependencies and build the dashboard:
209
206
210
207
.. code-block:: bash
211
208
212
209
npm ci
213
210
npm run build
214
211
215
-
After that, you can now move back to the toplevel Ray directory:
212
+
Move back to the top-level Ray directory:
216
213
217
214
.. code-block:: shell
218
215
@@ -236,20 +233,21 @@ Enter into the ``python/`` directory inside of the Ray project directory and ins
236
233
pip install -e . --verbose # Add --user if you see a permission denied error.
237
234
238
235
The ``-e`` means "editable", so changes you make to files in the Ray
239
-
directory will take effect without reinstalling the package.
236
+
directory take effect without reinstalling the package.
240
237
241
-
.. warning:: if you run ``python setup.py install``, files will be copied from the Ray directory to a directory of Python packages (``/lib/python3.6/site-packages/ray``). This means that changes you make to files in the Ray directory will not have any effect.
238
+
.. warning:: Don't run ``python setup.py install`` — Python copies files from the Ray directory to a packages directory (``/lib/python3.6/site-packages/ray``), so changes you make to files in the Ray directory won't have any effect.
242
239
243
240
.. tip::
244
241
245
-
If your machine is running out of memory during the build or the build is causing other programs to crash, try adding the following line to ``~/.bazelrc``:
242
+
If your machine runs out of memory during the build, add the following to ``~/.bazelrc``:
- Microsoft Visual Studio 2019 (or Microsoft Build Tools 2019 - https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019)
@@ -292,10 +290,10 @@ You can also use the included script to install Bazel:
292
290
293
291
3. Define an environment variable ``BAZEL_SH`` to point to ``bash.exe``. If git for Windows was installed for all users, bash's path should be ``C:\Program Files\Git\bin\bash.exe``. If git was installed for a single user, adjust the path accordingly.
294
292
295
-
4. Bazel 6.5.0 installation. Go to Bazel 6.5.0 release web page and download
296
-
bazel-4.2.1-windows-x86_64.exe. Copy the exe into the directory of your choice.
297
-
Define an environment variable BAZEL_PATH to full exe path (example:
298
-
``set BAZEL_PATH=C:\bazel\bazel.exe``). Also add the Bazel directory to the
293
+
4. Install Bazel 6.5.0. Go to the Bazel 6.5.0 release page and download
294
+
``bazel-6.5.0-windows-x86_64.exe``. Copy the exe into the directory of your choice.
295
+
Define an environment variable ``BAZEL_PATH`` to the full exe path (example:
296
+
``set BAZEL_PATH=C:\bazel\bazel.exe``). Also add the Bazel directory to
299
297
``PATH`` (example: ``set PATH=%PATH%;C:\bazel``)
300
298
301
299
5. Download ray source code and build it.
@@ -307,29 +305,28 @@ Define an environment variable BAZEL_PATH to full exe path (example:
307
305
cd ray\python
308
306
pip install -e . --verbose
309
307
310
-
Advanced Build Options
308
+
Advanced build options
311
309
----------------------
312
310
313
311
Environment variables that influence builds
314
312
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
315
313
316
314
You can tweak the build with the following environment variables (when running ``pip install -e .`` or ``python setup.py install``):
317
315
318
-
- ``RAY_BUILD_CORE``: If set and equal to ``1``, the core parts will be built. Defaults to ``1``.
319
-
- ``RAY_INSTALL_JAVA``: If set and equal to ``1``, extra build steps will be executed
320
-
to build java portions of the codebase
321
-
- ``RAY_INSTALL_CPP``: If set and equal to ``1``, ``ray-cpp`` will be installed
322
-
- ``RAY_BUILD_REDIS``: If set and equal to ``1``, Redis binaries will be built or fetched.
316
+
- ``RAY_BUILD_CORE``: If set and equal to ``1``, Ray builds the core parts. Defaults to ``1``.
317
+
- ``RAY_INSTALL_JAVA``: If set and equal to ``1``, Ray runs extra build steps
318
+
to build Java portions of the codebase
319
+
- ``RAY_INSTALL_CPP``: If set and equal to ``1``, Ray installs ``ray-cpp``
320
+
- ``RAY_BUILD_REDIS``: If set and equal to ``1``, Ray builds or fetches Redis binaries.
323
321
These binaries are only used for testing. Defaults to ``1``.
324
-
- ``RAY_DISABLE_EXTRA_CPP``: If set and equal to ``1``, a regular (non -
325
-
``cpp``) build will not provide some ``cpp`` interfaces
326
-
- ``SKIP_BAZEL_BUILD``: If set and equal to ``1``, no Bazel build steps will be
327
-
executed
328
-
- ``SKIP_THIRDPARTY_INSTALL_CONDA_FORGE``: If set, setup will skip installation of
322
+
- ``RAY_DISABLE_EXTRA_CPP``: If set and equal to ``1``, a regular (non-``cpp``)
323
+
build won't provide some ``cpp`` interfaces
324
+
- ``SKIP_BAZEL_BUILD``: If set and equal to ``1``, Ray skips all Bazel build steps
325
+
- ``SKIP_THIRDPARTY_INSTALL_CONDA_FORGE``: If set, setup skips installation of
329
326
third-party packages required for build. This is active on conda-forge where
330
-
pip is not used to create a build environment.
331
-
- ``RAY_DEBUG_BUILD``: Can be set to ``debug``, ``asan``, or ``tsan``. Any
332
-
other value will be ignored
327
+
pip isn't used to create a build environment.
328
+
- ``RAY_DEBUG_BUILD``: Can be set to ``debug``, ``asan``, or ``tsan``. Ray ignores
329
+
any other value
333
330
- ``BAZEL_ARGS``: If set, pass a space-separated set of arguments to Bazel. This can be useful
334
331
for restricting resource usage during builds, for example. See https://bazel.build/docs/user-manual
335
332
for more information about valid arguments.
@@ -340,19 +337,19 @@ You can tweak the build with the following environment variables (when running `
340
337
- ``BAZEL_PATH``: used on Windows to find ``bazel.exe``, see below
341
338
- ``MINGW_DIR``: used on Windows to find ``bazel.exe`` if not found in ``BAZEL_PATH``
342
339
343
-
Fast, Debug, and Optimized Builds
344
-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
340
+
Fast, debug, and optimized builds
341
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
345
342
346
-
Currently, Ray is built with optimizations, which can take a long time and
347
-
interfere with debugging. To perform fast, debug, or optimized builds, you can
343
+
By default, Ray builds with optimizations, which can take a long time and
344
+
interfere with debugging. To perform fast, debug, or optimized builds,
348
345
run the following (via ``-c`` ``fastbuild``/``dbg``/``opt``, respectively):
349
346
350
347
.. code-block:: shell
351
348
352
349
bazel run -c fastbuild //:gen_ray_pkg
353
350
354
-
This will rebuild Ray with the appropriate options (which may take a while).
355
-
If you need to build all targets, you can use ``bazel build //:all`` instead of
351
+
This rebuilds Ray with the appropriate options (which might take a while).
352
+
If you need to build all targets, use ``bazel build //:all`` instead of
356
353
``bazel run //:gen_ray_pkg``.
357
354
358
355
To make this change permanent, you can add an option such as the following
@@ -396,28 +393,28 @@ To test switching back to the original rule, change ``False`` to ``True``.
Requirement files for running Ray Data / ML library tests are under ``python/requirements/``.
419
416
420
-
Pre-commit Hooks
417
+
Pre-commit hooks
421
418
~~~~~~~~~~~~~~~~
422
419
423
420
Ray uses pre-commit hooks with `the pre-commit python package <https://pre-commit.com/>`_.
@@ -429,7 +426,7 @@ To start using ``pre-commit``:
429
426
pip install pre-commit
430
427
pre-commit install
431
428
432
-
This will install pre-commit into the current environment, and enable pre-commit checks every time
429
+
This installs pre-commit into the current environment and enables pre-commit checks every time
433
430
you commit new code changes with git. To temporarily skip pre-commit checks, use the ``-n`` or
434
431
``--no-verify`` flag when committing:
435
432
@@ -442,7 +439,7 @@ If you encounter any issues with ``pre-commit``, please `report an issue here`_.
442
439
.. _report an issue here: https://github.com/ray-project/ray/issues/new?template=bug-report.yml
443
440
444
441
445
-
Building the Docs
442
+
Building the docs
446
443
-----------------
447
444
448
445
To learn more about building the docs refer to `Contributing to the Ray Documentation`_.
@@ -472,7 +469,7 @@ in this error:
472
469
from . import _psutil_osx as cext
473
470
ImportError: cannot import name '_psutil_osx'from partially initialized module 'psutil' (most likely due to a circular import) (.../ray/python/ray/thirdparty_files/psutil/__init__.py)
0 commit comments