Databricks Runtime 8.3 for ML (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in June 2021.

Databricks Runtime 8.3 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 8.3 (EoS). Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. It also supports distributed deep learning training using Horovod.

For more information, including instructions for creating a Databricks Runtime ML cluster, see AI and machine learning on Databricks.

New features and improvements

Databricks Runtime 8.3 ML is built on top of Databricks Runtime 8.3. For information on what’s new in Databricks Runtime 8.3, including Apache Spark MLlib and SparkR, see the Databricks Runtime 8.3 (EoS) release notes.

Databricks Runtime 8.3 ML also includes the following new packages:

Major changes to Databricks Runtime ML Python environment

See Databricks Runtime 8.3 (EoS) for the major changes to the Databricks Runtime Python environment. For a full list of installed Python packages and their versions, see Python libraries.

Python packages upgraded

  • koalas 1.7.0 -> 1.8.0

  • mlflow 1.15.0 -> 1.17.0

  • pandas 1.1.3 -> 1.1.5

  • petastorm 0.9.8 -> 0.10.0

  • xgboost 1.3.3 -> 1.4.1

Python packages added

  • holidays: 0.10.5.2

Use Shiny inside R notebooks

You can now develop, host, and share Shiny applications directly from a Databricks R notebook, similarly to hosted RStudio. For details, see Shiny on Databricks.

Deprecations

Conda environments, along with the %conda command, are now deprecated in favor of pip and virtualenv and will be removed in an upcoming major release. Additionally, custom images using Conda-based environments with Databricks Container Services will still be supported, but will not have notebook-scoped library capabilities. Databricks recommends using virtualenv-based environments with Databricks Container Services and %pip for all notebook-scoped libraries.

System environment

The system environment in Databricks Runtime 8.3 ML differs from Databricks Runtime 8.3 as follows:

Libraries

The following sections list the libraries included in Databricks Runtime 8.3 ML that differ from those included in Databricks Runtime 8.3.

Python libraries

Databricks Runtime 8.3 ML uses Conda for Python package management and includes many popular ML packages.

In addition to the packages specified in the Conda environments in the following sections, Databricks Runtime 8.3 ML also includes the following packages:

  • hyperopt 0.2.5.db1

  • sparkdl 2.1.0.db4

  • feature_store 0.3.1

  • automl 1.0.0

Python libraries on CPU clusters 

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cpuonly=1.0=0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - ffmpeg=4.2.2=h20bf706_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - gmp=6.1.2=h6c8ec71_1
  - gnutls=3.6.15=he1e5248_0
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - h5py=2.10.0=py38h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libidn2=2.3.0=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.1.0=h2733197_1
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.2=hbbd107a_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - ninja=1.10.2=hff7bd54_1
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytorch=1.8.1=py3.8_cpu_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.9.1=py38_cpu
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - confuse==1.4.0
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - gast==0.3.3
    - grpcio==1.32.0
    - hijri-converter==2.1.1
    - holidays==0.10.5.2
    - horovod==0.21.3
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-preprocessing==1.1.2
    - koalas==1.8.0
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.16.1
    - mlflow-skinny==1.17.0
    - msrest==0.6.21
    - nbclient==0.5.3
    - nbconvert==6.0.7
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==2.11.0
    - pandocfilters==1.4.3
    - petastorm==0.10.0
    - phik==0.11.2
    - prometheus-client==0.10.1
    - pyarrow==1.0.1
    - pymeeus==0.5.11
    - pyrsistent==0.17.3
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.5.0
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.4.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow-cpu==2.4.1
    - tensorflow-estimator==2.4.0
    - termcolor==1.1.0
    - terminado==0.9.5
    - testpath==0.5.0
    - visions==0.6.0
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.1
prefix: /databricks/conda/envs/databricks-ml

Python libraries on GPU clusters 

name: databricks-ml-gpu
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.31.0=py38hf8bcb03_0
  - gunicorn=20.0.4=py38h06a4308_0
  - h5py=2.10.0=py38h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
    - argon2-cffi==20.1.0
    - astunparse==1.6.3
    - async-generator==1.10
    - azure-core==1.11.0
    - azure-storage-blob==12.7.1
    - bleach==3.3.0
    - confuse==1.4.0
    - convertdate==2.3.2
    - databricks-cli==0.14.3
    - defusedxml==0.7.1
    - diskcache==5.2.1
    - docker==4.4.4
    - facets-overview==1.0.0
    - flatbuffers==1.12
    - gast==0.3.3
    - hijri-converter==2.1.1
    - holidays==0.10.5.2
    - horovod==0.21.3
    - htmlmin==0.1.12
    - imagehash==4.2.0
    - ipywidgets==7.6.3
    - joblibspark==0.3.0
    - jsonschema==3.2.0
    - jupyterlab-pygments==0.1.2
    - jupyterlab-widgets==1.0.0
    - keras-preprocessing==1.1.2
    - koalas==1.8.0
    - korean-lunar-calendar==0.2.1
    - llvmlite==0.36.0
    - missingno==0.4.2
    - mistune==0.8.4
    - mleap==0.16.1
    - mlflow-skinny==1.17.0
    - msrest==0.6.21
    - nbclient==0.5.3
    - nbconvert==6.0.7
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.0
    - numba==0.53.1
    - opt-einsum==3.3.0
    - pandas-profiling==2.11.0
    - pandocfilters==1.4.3
    - petastorm==0.10.0
    - phik==0.11.2
    - pyarrow==1.0.1
    - pymeeus==0.5.11
    - pyrsistent==0.17.3
    - pywavelets==1.1.1
    - pyyaml==5.4.1
    - querystring-parser==1.2.4
    - seaborn==0.10.0
    - send2trash==1.5.0
    - shap==0.39.0
    - slicer==0.0.7
    - spark-tensorflow-distributor==0.1.0
    - tangled-up-in-unicode==0.1.0
    - tensorboard==2.4.1
    - tensorboard-plugin-wit==1.8.0
    - tensorflow==2.4.1
    - tensorflow-estimator==2.4.0
    - termcolor==1.1.0
    - terminado==0.9.5
    - testpath==0.5.0
    - torch==1.8.1
    - torchvision==0.9.1
    - visions==0.6.0
    - webencodings==0.5.1
    - widgetsnbextension==3.5.1
    - xgboost==1.4.1
prefix: /databricks/conda/envs/databricks-ml-gpu

Spark packages containing Python modules

Spark Package

Python Module

Version

graphframes

graphframes

0.8.1-db3-spark3.1

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 8.3.

Java and Scala libraries (Scala 2.12 cluster)

In addition to Java and Scala libraries in Databricks Runtime 8.3, Databricks Runtime 8.3 ML contains the following JARs:

CPU clusters

Group ID

Artifact ID

Version

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.combust.mleap

mleap-databricks-runtime_2.12

0.17.3-4882dc3

ml.dmlc

xgboost4j-spark_2.12

1.4.1

ml.dmlc

xgboost4j_2.12

1.4.1

org.mlflow

mlflow-client

1.17.0

org.scala-lang.modules

scala-java8-compat_2.12

0.8.0

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0

GPU clusters

Group ID

Artifact ID

Version

com.typesafe.akka

akka-actor_2.12

2.5.23

ml.combust.mleap

mleap-databricks-runtime_2.12

0.17.3-4882dc3

ml.dmlc

xgboost4j-spark-gpu_2.12

1.4.1

ml.dmlc

xgboost4j-gpu_2.12

1.4.1

org.mlflow

mlflow-client

1.17.0

org.scala-lang.modules

scala-java8-compat_2.12

0.8.0

org.tensorflow

spark-tensorflow-connector_2.12

1.15.0