Prepare data and environment for ML and DL
This section describes how to prepare your data and your Databricks environment for machine learning and deep learning.
The articles in this section cover aspects of loading and preprocessing data that are specific to ML and DL applications.
The Introduction to Databricks Runtime for Machine Learning (Databricks Runtime ML) is a ready-to-go environment optimized for machine learning and data science. Databricks Runtime ML includes many external libraries, including TensorFlow, PyTorch, Horovod, scikit-learn and XGBoost, and provides extensions to improve performance, including GPU acceleration in XGBoost, distributed deep learning using HorovodRunner, and model checkpointing using a Databricks File System (DBFS) FUSE mount.
To use Databricks Runtime ML, select the ML version of the runtime when you create your cluster.
To access data in Unity Catalog for machine learning workflows, you must use a Single user cluster. User Isolation clusters are not compatible with Databricks Runtime for Machine Learning.
You can install additional libraries to create a custom environment for your notebook or cluster.
To make a library available for all notebooks running on a cluster, create a cluster library. You can also use an init script to install libraries on clusters upon creation.
To install a library that is available only to a specific notebook session, use Notebook-scoped Python libraries.
Use GPU clusters
You can create GPU clusters to accelerate deep learning tasks. For information about creating Databricks GPU clusters, see GPU-enabled clusters. Databricks Runtime ML includes GPU hardware drivers and NVIDIA libraries such as CUDA.