This page describes the open source Deep Learning Pipelines package included in Databricks Runtime 6.6 ML and below. This page is not intended as a resource for general information about deep learning pipelines on Databricks.
The Deep Learning Pipelines package is a high-level deep learning framework that facilitates common deep learning workflows via the Apache Spark MLlib Pipelines API and scales out deep learning on big data using Spark. It is an open source project employing the Apache 2.0 License.
The Deep Learning Pipelines package calls into lower-level deep learning libraries. It supports TensorFlow and Keras with the TensorFlow backend.
Parts of the Deep Learning Pipelines library
sparkdl have been removed in Databricks Runtime 7.0 ML, specifically, the Transformers and Estimators used in Apache Spark ML pipelines. See the following sections for migration tips and workarounds.
The Deep Learning Pipelines package includes an image reader
sparkdl.image.imageIO, which was removed in Databricks Runtime 7.0 ML.
The Deep Learning Pipelines package includes a Spark ML Transformer
sparkdl.DeepImageFeaturizer for facilitating transfer learning with deep learning models.
DeepImageFeaturizer was removed in Databricks Runtime 7.0 ML.
Instead, use pandas UDFs to perform featurization with deep learning models. pandas UDFs, and their newer variant Scalar Iterator pandas UDFs, offer more flexible APIs, support more deep learning libraries, and give higher performance.
Refer to Featurization for transfer learning for examples of transfer learning with pandas UDFs.
The Deep Learning Pipelines package includes a Spark ML Estimator
sparkdl.KerasImageFileEstimator for tuning hyperparameters using Spark ML tuning utilities.
KerasImageFileEstimator was removed in Databricks Runtime 7.0 ML.
Instead, use Hyperparameter tuning with Hyperopt to distribute hyperparameter tuning for deep learning models.
The Deep Learning Pipelines package includes several Spark ML Transformers for distributing inference, all of which are removed in Databricks Runtime 7.0 ML:
The Deep Learning Pipelines package includes a utility
sparkdl.udf.keras_image_model.registerKerasImageUDF for deploying a deep learning model as a UDF callable from Spark SQL.
registerKerasImageUDF was removed in Databricks Runtime 7.0 ML.