ray-starter-notebook(Python)

Import Notebook

Ray on Databricks starter notebook

This notebook illustrates how to:

  • Create a Ray cluster on Databricks
  • Run a simple Ray application
  • Shut down a Ray cluster

Requirements

Databricks Runtime 12.0 or above.

Install Ray

%pip install ray[default]>=2.3.0
Python interpreter will be restarted. Collecting ray[default]@ https://ml-team-public-read.s3.us-west-2.amazonaws.com/ray-pkgs/demo0118/ray-3.0.0.dev0-cp39-cp39-linux_x86_64.whl Using cached https://ml-team-public-read.s3.us-west-2.amazonaws.com/ray-pkgs/demo0118/ray-3.0.0.dev0-cp39-cp39-linux_x86_64.whl (28.9 MB) Collecting msgpack<2.0.0,>=1.0.0 Using cached msgpack-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB) Collecting grpcio>=1.32.0 Using cached grpcio-1.51.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB) Collecting filelock Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB) Collecting attrs Using cached attrs-22.2.0-py3-none-any.whl (60 kB) Collecting aiosignal Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting virtualenv>=20.0.24 Using cached virtualenv-20.17.1-py3-none-any.whl (8.8 MB) Collecting frozenlist Using cached frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB) Collecting click>=7.0 Using cached click-8.1.3-py3-none-any.whl (96 kB) Collecting numpy>=1.19.3 Using cached numpy-1.24.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB) Collecting protobuf!=3.19.5,>=3.15.3 Using cached protobuf-4.21.12-cp37-abi3-manylinux2014_x86_64.whl (409 kB) Collecting requests Using cached requests-2.28.2-py3-none-any.whl (62 kB) Collecting pyyaml Using cached PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB) Collecting jsonschema Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB) Collecting aiohttp-cors Using cached aiohttp_cors-0.7.0-py3-none-any.whl (27 kB) Collecting prometheus-client>=0.7.1 Using cached prometheus_client-0.15.0-py3-none-any.whl (60 kB) Collecting pydantic Using cached pydantic-1.10.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB) Collecting opencensus Using cached opencensus-0.11.0-py2.py3-none-any.whl (128 kB) Collecting smart-open Using cached smart_open-6.3.0-py3-none-any.whl (56 kB) Collecting aiohttp>=3.7 Using cached aiohttp-3.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB) Collecting gpustat>=1.0.0 Using cached gpustat-1.0.0-py3-none-any.whl Collecting py-spy>=0.2.0 Using cached py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB) Collecting colorful Using cached colorful-0.5.5-py2.py3-none-any.whl (201 kB) Collecting charset-normalizer<3.0,>=2.0 Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB) Collecting yarl<2.0,>=1.0 Using cached yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB) Collecting async-timeout<5.0,>=4.0.0a3 Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting multidict<7.0,>=4.5 Using cached multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB) Collecting six>=1.7 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting nvidia-ml-py<=11.495.46,>=11.450.129 Using cached nvidia_ml_py-11.495.46-py3-none-any.whl (25 kB) Collecting psutil>=5.6.0 Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) Collecting blessed>=1.17.1 Using cached blessed-1.19.1-py2.py3-none-any.whl (58 kB) Collecting wcwidth>=0.1.4 Using cached wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) Collecting platformdirs<3,>=2.4 Using cached platformdirs-2.6.2-py3-none-any.whl (14 kB) Collecting distlib<1,>=0.3.6 Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB) Collecting idna>=2.0 Using cached idna-3.4-py3-none-any.whl (61 kB) Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 Using cached pyrsistent-0.19.3-py3-none-any.whl (57 kB) Collecting google-api-core<3.0.0,>=1.0.0 Using cached google_api_core-2.11.0-py3-none-any.whl (120 kB) Collecting opencensus-context>=0.1.3 Using cached opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB) Collecting googleapis-common-protos<2.0dev,>=1.56.2 Using cached googleapis_common_protos-1.58.0-py2.py3-none-any.whl (223 kB) Collecting google-auth<3.0dev,>=2.14.1 Using cached google_auth-2.16.0-py2.py3-none-any.whl (177 kB) Collecting cachetools<6.0,>=2.0.0 Using cached cachetools-5.2.1-py3-none-any.whl (9.3 kB) Collecting rsa<5,>=3.1.4 Using cached rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting pyasn1<0.5.0,>=0.4.6 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB) Collecting certifi>=2017.4.17 Using cached certifi-2022.12.7-py3-none-any.whl (155 kB) Collecting typing-extensions>=4.2.0 Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB) Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, multidict, idna, frozenlist, charset-normalizer, certifi, cachetools, yarl, wcwidth, requests, pyrsistent, platformdirs, googleapis-common-protos, google-auth, filelock, distlib, attrs, async-timeout, aiosignal, virtualenv, typing-extensions, pyyaml, psutil, opencensus-context, nvidia-ml-py, numpy, msgpack, jsonschema, grpcio, google-api-core, click, blessed, aiohttp, smart-open, ray, pydantic, py-spy, prometheus-client, opencensus, gpustat, colorful, aiohttp-cors Attempting uninstall: pyasn1 Found existing installation: pyasn1 0.4.8 Uninstalling pyasn1-0.4.8: Successfully uninstalled pyasn1-0.4.8 Attempting uninstall: urllib3 Found existing installation: urllib3 1.26.14 Uninstalling urllib3-1.26.14: Successfully uninstalled urllib3-1.26.14 Attempting uninstall: six Found existing installation: six 1.16.0 Uninstalling six-1.16.0: Successfully uninstalled six-1.16.0 Attempting uninstall: rsa Found existing installation: rsa 4.9 Uninstalling rsa-4.9: Successfully uninstalled rsa-4.9 Attempting uninstall: pyasn1-modules Found existing installation: pyasn1-modules 0.2.8 Uninstalling pyasn1-modules-0.2.8: Successfully uninstalled pyasn1-modules-0.2.8 Attempting uninstall: protobuf Found existing installation: protobuf 4.21.12 Uninstalling protobuf-4.21.12: Successfully uninstalled protobuf-4.21.12 Attempting uninstall: multidict Found existing installation: multidict 6.0.4 Uninstalling multidict-6.0.4: Successfully uninstalled multidict-6.0.4 Attempting uninstall: idna Found existing installation: idna 3.4 Uninstalling idna-3.4: Successfully uninstalled idna-3.4 Attempting uninstall: frozenlist Found existing installation: frozenlist 1.3.3 Uninstalling frozenlist-1.3.3: Successfully uninstalled frozenlist-1.3.3 Attempting uninstall: charset-normalizer Found existing installation: charset-normalizer 2.1.1 Uninstalling charset-normalizer-2.1.1: Successfully uninstalled charset-normalizer-2.1.1 Attempting uninstall: certifi Found existing installation: certifi 2022.12.7 Uninstalling certifi-2022.12.7: Successfully uninstalled certifi-2022.12.7 Attempting uninstall: cachetools Found existing installation: cachetools 5.2.1 Uninstalling cachetools-5.2.1: Successfully uninstalled cachetools-5.2.1 Attempting uninstall: yarl Found existing installation: yarl 1.8.2 Uninstalling yarl-1.8.2: Successfully uninstalled yarl-1.8.2 Attempting uninstall: wcwidth Found existing installation: wcwidth 0.2.6 Uninstalling wcwidth-0.2.6: Successfully uninstalled wcwidth-0.2.6 Attempting uninstall: requests Found existing installation: requests 2.28.2 Uninstalling requests-2.28.2: Successfully uninstalled requests-2.28.2 Attempting uninstall: pyrsistent Found existing installation: pyrsistent 0.19.3 Uninstalling pyrsistent-0.19.3: Successfully uninstalled pyrsistent-0.19.3 Attempting uninstall: platformdirs Found existing installation: platformdirs 2.6.2 Uninstalling platformdirs-2.6.2: Successfully uninstalled platformdirs-2.6.2 Attempting uninstall: googleapis-common-protos Found existing installation: googleapis-common-protos 1.58.0 Uninstalling googleapis-common-protos-1.58.0: Successfully uninstalled googleapis-common-protos-1.58.0 Attempting uninstall: google-auth Found existing installation: google-auth 2.16.0 Uninstalling google-auth-2.16.0: Successfully uninstalled google-auth-2.16.0 Attempting uninstall: filelock Found existing installation: filelock 3.9.0 Uninstalling filelock-3.9.0: Successfully uninstalled filelock-3.9.0 Attempting uninstall: distlib Found existing installation: distlib 0.3.6 Uninstalling distlib-0.3.6: Successfully uninstalled distlib-0.3.6 Attempting uninstall: attrs Found existing installation: attrs 22.2.0 Uninstalling attrs-22.2.0: Successfully uninstalled attrs-22.2.0 Attempting uninstall: async-timeout Found existing installation: async-timeout 4.0.2 Uninstalling async-timeout-4.0.2: Successfully uninstalled async-timeout-4.0.2 Attempting uninstall: aiosignal Found existing installation: aiosignal 1.3.1 Uninstalling aiosignal-1.3.1: Successfully uninstalled aiosignal-1.3.1 Attempting uninstall: virtualenv Found existing installation: virtualenv 20.17.1 Uninstalling virtualenv-20.17.1: Successfully uninstalled virtualenv-20.17.1 Attempting uninstall: typing-extensions Found existing installation: typing-extensions 4.4.0 Uninstalling typing-extensions-4.4.0: Successfully uninstalled typing-extensions-4.4.0 Attempting uninstall: pyyaml Found existing installation: PyYAML 6.0 Uninstalling PyYAML-6.0: Successfully uninstalled PyYAML-6.0 Attempting uninstall: psutil Found existing installation: psutil 5.9.4 Uninstalling psutil-5.9.4: Successfully uninstalled psutil-5.9.4 Attempting uninstall: opencensus-context Found existing installation: opencensus-context 0.1.3 Uninstalling opencensus-context-0.1.3: Successfully uninstalled opencensus-context-0.1.3 Attempting uninstall: nvidia-ml-py Found existing installation: nvidia-ml-py 11.495.46 Uninstalling nvidia-ml-py-11.495.46: Successfully uninstalled nvidia-ml-py-11.495.46 Attempting uninstall: numpy Found existing installation: numpy 1.24.1 Uninstalling numpy-1.24.1: Successfully uninstalled numpy-1.24.1 Attempting uninstall: msgpack Found existing installation: msgpack 1.0.4 Uninstalling msgpack-1.0.4: Successfully uninstalled msgpack-1.0.4 Attempting uninstall: jsonschema Found existing installation: jsonschema 4.17.3 Uninstalling jsonschema-4.17.3: Successfully uninstalled jsonschema-4.17.3 Attempting uninstall: grpcio Found existing installation: grpcio 1.51.1 Uninstalling grpcio-1.51.1: Successfully uninstalled grpcio-1.51.1 Attempting uninstall: google-api-core Found existing installation: google-api-core 2.11.0 Uninstalling google-api-core-2.11.0: Successfully uninstalled google-api-core-2.11.0 Attempting uninstall: click Found existing installation: click 8.1.3 Uninstalling click-8.1.3: Successfully uninstalled click-8.1.3 Attempting uninstall: blessed Found existing installation: blessed 1.19.1 Uninstalling blessed-1.19.1: Successfully uninstalled blessed-1.19.1 Attempting uninstall: aiohttp Found existing installation: aiohttp 3.8.3 Uninstalling aiohttp-3.8.3: Successfully uninstalled aiohttp-3.8.3 Attempting uninstall: smart-open Found existing installation: smart-open 6.3.0 Uninstalling smart-open-6.3.0: Successfully uninstalled smart-open-6.3.0 Attempting uninstall: ray Found existing installation: ray 3.0.0.dev0 Uninstalling ray-3.0.0.dev0: Successfully uninstalled ray-3.0.0.dev0 Attempting uninstall: pydantic Found existing installation: pydantic 1.10.4 Uninstalling pydantic-1.10.4: Successfully uninstalled pydantic-1.10.4 Attempting uninstall: py-spy Found existing installation: py-spy 0.3.14 Uninstalling py-spy-0.3.14: Successfully uninstalled py-spy-0.3.14 Attempting uninstall: prometheus-client Found existing installation: prometheus-client 0.15.0 Uninstalling prometheus-client-0.15.0: Successfully uninstalled prometheus-client-0.15.0 Attempting uninstall: opencensus Found existing installation: opencensus 0.11.0 Uninstalling opencensus-0.11.0: Successfully uninstalled opencensus-0.11.0 Attempting uninstall: gpustat Found existing installation: gpustat 1.0.0 Uninstalling gpustat-1.0.0: Successfully uninstalled gpustat-1.0.0 Attempting uninstall: colorful Found existing installation: colorful 0.5.5 Uninstalling colorful-0.5.5: Successfully uninstalled colorful-0.5.5 Attempting uninstall: aiohttp-cors Found existing installation: aiohttp-cors 0.7.0 Uninstalling aiohttp-cors-0.7.0: Successfully uninstalled aiohttp-cors-0.7.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. petastorm 0.12.0 requires pyspark>=2.1.0, which is not installed. databricks-feature-store 0.9.0 requires pyspark<4,>=3.1.2, which is not installed. tensorflow-cpu 2.10.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.21.12 which is incompatible. tensorboard 2.10.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.21.12 which is incompatible. scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.24.1 which is incompatible. pathy 0.6.1 requires smart-open<6.0.0,>=5.0.0, but you have smart-open 6.3.0 which is incompatible. pandas-profiling 3.5.0 requires numpy<1.24,>=1.16.0, but you have numpy 1.24.1 which is incompatible. numba 0.55.1 requires numpy<1.22,>=1.18, but you have numpy 1.24.1 which is incompatible. mleap 0.20.0 requires scikit-learn<0.23.0,>=0.22.0, but you have scikit-learn 1.0.2 which is incompatible. Successfully installed aiohttp-3.8.3 aiohttp-cors-0.7.0 aiosignal-1.3.1 async-timeout-4.0.2 attrs-22.2.0 blessed-1.19.1 cachetools-5.2.1 certifi-2022.12.7 charset-normalizer-2.1.1 click-8.1.3 colorful-0.5.5 distlib-0.3.6 filelock-3.9.0 frozenlist-1.3.3 google-api-core-2.11.0 google-auth-2.16.0 googleapis-common-protos-1.58.0 gpustat-1.0.0 grpcio-1.51.1 idna-3.4 jsonschema-4.17.3 msgpack-1.0.4 multidict-6.0.4 numpy-1.24.1 nvidia-ml-py-11.495.46 opencensus-0.11.0 opencensus-context-0.1.3 platformdirs-2.6.2 prometheus-client-0.15.0 protobuf-4.21.12 psutil-5.9.4 py-spy-0.3.14 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydantic-1.10.4 pyrsistent-0.19.3 pyyaml-6.0 ray-3.0.0.dev0 requests-2.28.2 rsa-4.9 six-1.16.0 smart-open-6.3.0 typing-extensions-4.4.0 urllib3-1.26.14 virtualenv-20.17.1 wcwidth-0.2.6 yarl-1.8.2 Python interpreter will be restarted.

Create a Ray cluster

Set up a Ray cluster with 2 Ray worker nodes. Each worker node is assigned with 4 GPU cores. Once the cluster is launched, you can click the link "Open Ray Cluster Dashboard in a new tab" to view the Ray cluster dashboard.

from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster, MAX_NUM_WORKER_NODES


setup_ray_cluster(
  num_worker_nodes=2,
  num_cpus_per_node=4,
  collect_log_to_path="/dbfs/tmp/raylogs",
)
INFO:ray.util.spark:Ray head node started. 2023-01-18 10:35:44,400 INFO usage_lib.py:461 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. 2023-01-18 10:35:44,400 INFO scripts.py:710 -- Local node IP: 10.68.142.34 2023-01-18 10:35:48,046 SUCC scripts.py:747 -- -------------------- 2023-01-18 10:35:48,046 SUCC scripts.py:748 -- Ray runtime started. 2023-01-18 10:35:48,046 SUCC scripts.py:749 -- -------------------- 2023-01-18 10:35:48,047 INFO scripts.py:751 -- Next steps 2023-01-18 10:35:48,047 INFO scripts.py:752 -- To connect to this Ray runtime from another node, run 2023-01-18 10:35:48,047 INFO scripts.py:755 -- ray start --address='10.68.142.34:9818' 2023-01-18 10:35:48,047 INFO scripts.py:771 -- Alternatively, use the following Python code: 2023-01-18 10:35:48,047 INFO scripts.py:773 -- import ray 2023-01-18 10:35:48,047 INFO scripts.py:777 -- ray.init(address='auto', _node_ip_address='10.68.142.34') 2023-01-18 10:35:48,047 INFO scripts.py:789 -- To connect to this Ray runtime from outside of the cluster, for example to 2023-01-18 10:35:48,048 INFO scripts.py:793 -- connect to a remote cluster from your laptop directly, use the following 2023-01-18 10:35:48,048 INFO scripts.py:797 -- Python code: 2023-01-18 10:35:48,048 INFO scripts.py:799 -- import ray 2023-01-18 10:35:48,049 INFO scripts.py:800 -- ray.init(address='ray://<head_node_ip_address>:10001') 2023-01-18 10:35:48,049 INFO scripts.py:809 -- To see the status of the cluster, use 2023-01-18 10:35:48,051 INFO scripts.py:810 -- ray status 2023-01-18 10:35:48,051 INFO scripts.py:813 -- To monitor and debug Ray, view the dashboard at 2023-01-18 10:35:48,051 INFO scripts.py:814 -- 10.68.142.34:9436 2023-01-18 10:35:48,051 INFO scripts.py:820 -- If connection fails, check your firewall settings and network configuration. 2023-01-18 10:35:48,051 INFO scripts.py:828 -- To terminate the Ray runtime, run 2023-01-18 10:35:48,051 INFO scripts.py:829 -- ray stop 2023-01-18 10:35:48,051 INFO scripts.py:907 -- --block 2023-01-18 10:35:48,051 INFO scripts.py:908 -- This command will now block forever until terminated by a signal. 2023-01-18 10:35:48,052 INFO scripts.py:911 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported. 2023-01-18 10:36:17,955 INFO worker.py:1366 -- Connecting to existing Ray cluster at address: 10.68.142.34:9818... 2023-01-18 10:36:17,969 INFO worker.py:1546 -- Connected to Ray cluster. View the dashboard at 10.68.142.34:9436
Out[2]: '10.68.142.34:9818'

Show the resources of the Ray cluster you just created.

import ray
ray.init()
ray.cluster_resources()
2023-01-18 10:36:22,254 INFO worker.py:1244 -- Using address 10.68.142.34:9818 set in the environment variable RAY_ADDRESS 2023-01-18 10:36:22,255 INFO worker.py:1366 -- Connecting to existing Ray cluster at address: 10.68.142.34:9818... 2023-01-18 10:36:22,260 INFO worker.py:1546 -- Connected to Ray cluster. View the dashboard at 10.68.142.34:9436 Out[3]: {'CPU': 8.0, 'memory': 30076850668.0, 'object_store_memory': 12966774700.0, 'node:10.68.153.51': 1.0, 'node:10.68.142.34': 1.0, 'node:10.68.135.15': 1.0}

Run Ray application

The following code runs a simple Ray application that creates many Ray tasks on the Ray cluster you just created.

import ray
import random
import time
import math
from fractions import Fraction

@ray.remote
def pi4_sample(sample_count):
    """pi4_sample runs sample_count experiments, and returns the 
    fraction of time it was inside the circle. 
    """
    in_count = 0
    for i in range(sample_count):
        x = random.random()
        y = random.random()
        if x*x + y*y <= 1:
            in_count += 1
    return Fraction(in_count, sample_count)

SAMPLE_COUNT = 1000 * 1000
start = time.time() 
future = pi4_sample.remote(sample_count = SAMPLE_COUNT)
pi4 = ray.get(future)
end = time.time()
dur = end - start
print(f'Running {SAMPLE_COUNT} tests took {dur} seconds')

pi = pi4 * 4
print(float(pi))
Running 1000000 tests took 1.504802942276001 seconds 3.141832 
FULL_SAMPLE_COUNT = 2000 * 1000 * 1000
BATCHES = int(FULL_SAMPLE_COUNT / SAMPLE_COUNT)
print(f'Doing {BATCHES} batches')
results = []
for _ in range(BATCHES):
    results.append(pi4_sample.remote(sample_count = SAMPLE_COUNT))
output = ray.get(results)

pi = sum(output)*4/len(output)
print(float(pi))
Doing 2000 batches 3.141669298

Shut down the Ray cluster you just created.

shutdown_ray_cluster()