Databricks provides an environment that makes it easy to build, train, and deploy deep learning models at scale. Many deep learning libraries are available in Databricks Runtime ML, a machine learning runtime that provides a ready-to-go environment for machine learning and data science. For deep learning libraries not included in Databricks Runtime ML, you can either install libraries as a Databricks library or use init scripts to install libraries on clusters upon creation.
Graphics processing units (GPUs) can accelerate deep learning tasks. For information about creating GPU-enabled Databricks clusters, see GPU-enabled clusters. Databricks Runtime ML includes installed GPU hardware drivers and NVIDIA libraries such as CUDA.
A typical deep learning workflow involves the phases data preparation, training, and inference. This section gives guidelines on deep learning in Databricks.
- Data preparation
- Single node training
- Distributed training
- Model inference
- Reference solutions