Databricks Runtime 5.5 ML

Databricks released this image in July 2019.

Databricks Runtime 5.5 ML provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 5.5. Databricks Runtime for ML contains many popular machine learning libraries, including TensorFlow, PyTorch, Keras, and XGBoost. It also supports distributed deep learning training using Horovod.

For more information, including instructions for creating a Databricks Runtime ML cluster, see Overview of Databricks Runtime for Machine Learning.

New features

Databricks Runtime 5.5 ML is built on top of Databricks Runtime 5.5. For information on what’s new in Databricks Runtime 5.5, see the Databricks Runtime 5.5 release notes.

In addition to library updates, Databricks Runtime 5.5 ML introduces the following new features:

  • Added the MLflow 1.0 Python package

Improvements

  • Upgraded machine learning libraries

    • Tensorflow upgraded from 1.12.0 to 1.13.1
    • PyTorch upgraded from 0.4.1 to 1.1.0
    • scikit-learn upgraded from 0.19.1 to 0.20.3
  • Single-node operation for HorovodRunner

    Enabled HorovodRunner to run on only the driver node. Previously, to use HorovodRunner you would have to run a driver and at least one worker node. With this change, you can now distribute training within a single node (that is, a multi-GPU node) and thus use compute resources more efficiently.

Deprecation

In the hyperopt library, we deprecated the following properties of hyperopt.SparkTrials:

  • SparkTrials.successful_trials_count
  • SparkTrials.failed_trials_count
  • SparkTrials.cancelled_trials_count
  • SparkTrials.total_trials_count

and replaced the properties with the following functions:

  • SparkTrials.count_successful_trials()
  • SparkTrials.count_failed_trials()
  • SparkTrials.count_cancelled_trials()
  • SparkTrials.count_total_trials()

System environment

The system environment in Databricks Runtime 5.5 ML differs from Databricks Runtime 5.5 as follows:

  • Python: 3.6.5 for Python 3 clusters and 2.7.15 for Python 2 clusters.
  • DBUtils: Databricks Runtime 5.5 ML does not contain Library utilities.
  • For GPU clusters, the following NVIDIA GPU libraries:
    • Tesla driver 418.40
    • CUDA 10.0
    • CUDNN 7.6.0

Libraries

The following sections list the libraries included in Databricks Runtime 5.5 ML that differ from those included in Databricks Runtime 5.5.

Python libraries

Databricks Runtime 5.5 ML uses Conda for Python package management. As a result, there are major differences in installed Python libraries compared to Databricks Runtime. The following sections describe the Conda environments for Databricks Runtime 5.5 ML clusters using Python 2 or 3, and CPU or GPU-enabled machines.

Python 3 on CPU clusters

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36hb0f60f5_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36h17d85b1_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36he6710b0_0
  - py-xgboost-cpu=0.90=py36_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow=1.13.1=mkl_py36h27d456a_0
  - tensorflow-base=1.13.1=mkl_py36h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36h8cadb63_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py3.6_cpu_0
  - torchvision-cpu=0.3.0=py36_cuNone_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 3 on GPU clusters

name: null
channels:
  - pytorch
  - Databricks
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py36h2903d8e_0
  - tensorflow-base=1.13.1.db1=gpu_py36he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py36_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36hb52b0d5_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36h63277f8_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36h688424c_0
  - py-xgboost-gpu=0.90=py36h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36h7a266c3_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36h674d592_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py3.6_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py36_cu10.0.130_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 2 on CPU clusters

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27h5cde069_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cycler=0.10.0=py27hc7354d3_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27he6710b0_0
  - py-xgboost-cpu=0.90=py27_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow=1.13.1=mkl_py27h74ee40f_0
  - tensorflow-base=1.13.1=mkl_py27h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27hc38d2c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py2.7_cpu_0
  - torchvision-cpu=0.3.0=py27_cuNone_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Python 2 on GPU clusters

name: null
channels:
  - Databricks
  - pytorch
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py27h8e347d7_0
  - tensorflow-base=1.13.1.db1=gpu_py27he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py27_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27hae222c1_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27h75840f5_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27h09770e1_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27h688424c_0
  - py-xgboost-gpu=0.90=py27h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27hd6ce930_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py2.7_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py27_cu10.0.130_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Spark packages containing Python modules

Spark Package Python Module Version
graphframes graphframes 0.7.0-db1-spark2.4
spark-deep-learning sparkdl 1.5.0-db4-spark2.4
tensorframes tensorframes 0.7.0-s_2.11

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 5.5.

Java and Scala libraries (Scala 2.11 cluster)

In addition to Java and Scala libraries in Databricks Runtime 5.5, Databricks Runtime 5.5 ML contains the following JARs:

Group ID Artifact ID Version
com.databricks spark-deep-learning 1.5.0-db4-spark2.4
com.typesafe.akka akka-actor_2.11 2.3.11
ml.combust.mleap mleap-databricks-runtime_2.11 0.13.0
ml.dmlc xgboost4j 0.90
ml.dmlc xgboost4j-spark 0.90
org.graphframes graphframes_2.11 0.7.0-db1-spark2.4
org.tensorflow libtensorflow 1.13.1
org.tensorflow libtensorflow_jni 1.13.1
org.tensorflow spark-tensorflow-connector_2.11 1.13.1
org.tensorflow tensorflow 1.13.1
org.tensorframes tensorframes 0.7.0-s_2.11