Databricks Runtime 8.4 for ML (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in July 2021.

Databricks Runtime 8.4 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 8.4 (EoS). Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. It also supports distributed deep learning training using Horovod.

For more information, including instructions for creating a Databricks Runtime ML cluster, see AI and machine learning on Databricks.

New features and improvements

Databricks Runtime 8.4 ML is built on top of Databricks Runtime 8.4. For information on what’s new in Databricks Runtime 8.4, including Apache Spark MLlib and SparkR, see the Databricks Runtime 8.4 (EoS) release notes.

FeatureStoreClient v0.3.2

Allow feature and feature table names that conflict with SQL reserved words.
Validate that provided DataFrames are PySpark DataFrames (pyspark.sql.dataframe.DataFrame).

AutoML v1.1.0

The updated version of AutoML that ships with Databricks Runtime 8.4 ML includes some bug fixes and stability improvements.
AutoML Classification now also runs trials with LGBMClassifier
AutoML Regression now also runs trials with LGBMRegressor

Major changes to Databricks Runtime ML Python environment

See Databricks Runtime 8.4 (EoS) for the major changes to the Databricks Runtime Python environment. For a full list of installed Python packages and their versions, see Python libraries.

Python packages upgraded

koalas 1.8.0 -> 1.8.1
horovod 0.21.3 -> 0.22.1
mleap 0.16.1 -> 0.17.0
mlflow 1.16.0 -> 1.18.0
pandas-profiling 2.11.0 -> 3.0.0
petastorm 0.10.0 -> 0.11.1
pytorch 1.8.1 -> 1.9.0
tensorboard 2.4.1 -> 2.5.0
tensorflow 2.4.1 -> 2.5.0
torchvision 0.9.1 -> 0.10.0
xgboost 1.4.1 -> 1.4.2

Deprecations

The following changes are deprecated and will be removed in Databricks Runtime 9.0:

In HorovodRunner, setting np=0, where np is the number of parallel processes to use for the Horovod job.
Intel Math Kernel Library (Intel MKL), along with downstream flavors of packages that depend on it.
The azure-core python library for core Azure exceptions and modules
The azure-storage-blob python library client for interacting with Azure Storage Blob service
The msrest python library for AutoRest swagger generation
The docker python library for the Docker Engine API
The querystring-parser python library for parsing queries in Python/Django
The intel-openmp python library for creating multithreaded software

System environment

The system environment in Databricks Runtime 8.4 ML differs from Databricks Runtime 8.4 as follows:

DBUtils: Databricks Runtime ML does not include Library utility (dbutils.library) (legacy). Use %pip and %conda commands instead. See Notebook-scoped Python libraries.
For GPU clusters, Databricks Runtime ML includes the following NVIDIA GPU libraries:
- CUDA 11.0
- cuDNN 8.0.4.30
- NCCL 2.7.8
- TensorRT 7.1.3

Libraries

The following sections list the libraries included in Databricks Runtime 8.4 ML that differ from those included in Databricks Runtime 8.4.

In this section:

Top-tier libraries
Python libraries
R libraries
Java and Scala libraries (Scala 2.12 cluster)

Top-tier libraries

Databricks Runtime 8.4 ML includes the following top-tier libraries:

Python libraries

Databricks Runtime 8.4 ML uses Conda for Python package management and includes many popular ML packages.

In addition to the packages specified in the Conda environments in the following sections, Databricks Runtime 8.4 ML also includes the following packages:

hyperopt 0.2.5.db2
sparkdl 2.1.0.db4
feature_store 0.3.2
automl 1.1.0

Python libraries on CPU clusters

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2021.5.25=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cpuonly=1.0=0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - ffmpeg=4.2.2=h20bf706_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gast=0.4.0=py_0
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - gmp=6.1.2=h6c8ec71_1
  - gnutls=3.6.15=he1e5248_0
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libidn2=2.3.1=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.1.0=h2733197_1
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.3=hbbd107a_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - ninja=1.10.2=hff7bd54_1
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytorch=1.9.0=py3.8_cpu_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.10.0=py38_cpu
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - bottleneck==1.3.2
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - grpcio==1.34.1
    - h5py==3.1.0
    - hijri-converter==2.1.3
    - holidays==0.10.5.2
    - horovod==0.22.1
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-nightly==2.5.0.dev2021032900
    - keras-preprocessing==1.1.2
    - koalas==1.8.1
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.17.0
    - mlflow-skinny==1.18.0
    - msrest==0.6.21
    - multimethod==1.4
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==3.0.0
    - pandocfilters==1.4.3
    - petastorm==0.11.1
    - phik==0.11.2
    - prometheus-client==0.11.0
    - pyarrow==1.0.1
    - pydantic==1.8.2
    - pymeeus==0.5.11
    - pyrsistent==0.18.0
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.7.1
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.5.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow-cpu==2.5.0
    - tensorflow-estimator==2.5.0
    - termcolor==1.1.0
    - terminado==0.10.1
    - testpath==0.5.0
    - visions==0.7.1
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.2
prefix: /databricks/conda/envs/databricks-ml

Python libraries on GPU clusters

name: databricks-ml-gpu
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - ca-certificates=2021.5.25=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gast=0.4.0=py_0
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - bottleneck==1.3.2
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - grpcio==1.34.1
    - h5py==3.1.0
    - hijri-converter==2.1.3
    - holidays==0.10.5.2
    - horovod==0.22.1
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-nightly==2.5.0.dev2021032900
    - keras-preprocessing==1.1.2
    - koalas==1.8.1
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.17.0
    - mlflow-skinny==1.18.0
    - msrest==0.6.21
    - multimethod==1.4
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==3.0.0
    - pandocfilters==1.4.3
    - petastorm==0.11.1
    - phik==0.11.2
    - pyarrow==1.0.1
    - pydantic==1.8.2
    - pymeeus==0.5.11
    - pyrsistent==0.17.3
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.7.1
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.5.0
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow==2.5.0
    - tensorflow-estimator==2.5.0
    - termcolor==1.1.0
    - terminado==0.10.1
    - testpath==0.5.0
    - torch==1.9.0
    - torchvision==0.10.0
    - visions==0.7.1
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.2
prefix: /databricks/conda/envs/databricks-ml-gpu

Spark packages containing Python modules

Spark Package	Python Module	Version
graphframes	graphframes	0.8.1-db3-spark3.1

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 8.4.

Java and Scala libraries (Scala 2.12 cluster)

In addition to Java and Scala libraries in Databricks Runtime 8.4, Databricks Runtime 8.4 ML contains the following JARs:

CPU clusters

Group ID	Artifact ID	Version
com.typesafe.akka	akka-actor_2.12	2.5.23
ml.combust.mleap	mleap-databricks-runtime_2.12	0.17.3-4882dc3
ml.dmlc	xgboost4j-spark_2.12	1.4.1
ml.dmlc	xgboost4j_2.12	1.4.1
org.mlflow	mlflow-client	1.18.0
org.scala-lang.modules	scala-java8-compat_2.12	0.8.0
org.tensorflow	spark-tensorflow-connector_2.12	1.15.0

GPU clusters

Group ID	Artifact ID	Version
com.typesafe.akka	akka-actor_2.12	2.5.23
ml.combust.mleap	mleap-databricks-runtime_2.12	0.17.3-4882dc3
ml.dmlc	xgboost4j-spark-gpu_2.12	1.4.1
ml.dmlc	xgboost4j-gpu_2.12	1.4.1
org.mlflow	mlflow-client	1.18.0
org.scala-lang.modules	scala-java8-compat_2.12	0.8.0
org.tensorflow	spark-tensorflow-connector_2.12	1.15.0