This section provides a guide to developing notebooks and jobs in Databricks using the Python language.
This notebook will get you up to speed quickly on using the scikit-learn library on Databricks. It covers data loading and preparation; model training, tuning, and inference; and model deployment and management with MLflow.
PySpark is the Python API for Apache Spark. These links provide an introduction to and reference for PySpark.
Databricks Python notebooks support various types of visualizations using the
You can also use the following third-party libraries to create visualizations in Databricks Python notebooks.
These articles describe features that support interoperability between PySpark and pandas.
This article describes features that support interoperability between Python and SQL.
In addition to Databricks notebooks, you can use the following Python developer tools:
Databricks runtimes include many popular libraries. You can also install additional third-party or custom Python libraries to use with notebooks and jobs running on Databricks clusters.
Cluster-based libraries are available to all notebooks and jobs running on the cluster. For information about installing cluster-based libraries, see Install a library on a cluster.
Notebook-scoped libraries are available only to the notebook on which they are installed and must be reinstalled for each session.