Databricks for Python developers

This section provides a guide to developing notebooks and jobs in Databricks using the Python language.

Python API

The API that provides a Python interface to Apache Spark is PySpark. These links provide an introduction to and reference for PySpark.

• Introduction to DataFrames
• Introduction to Structured Streaming
• PySpark API reference

Visualizations

Databricks Python notebooks support various types of visualizations using the display function.

• Visualizations in Python
• Visualization deep dive in Python

You can also use the following third-party libraries to create visualizations in Databricks Python notebooks.

• Bokeh in Python notebooks
• Matplotlib plots in Python notebooks
• Plotly in Python notebooks

Interoperability

This section describes features that support interoperability between Python and SQL.

• User-defined functions - Python
• pandas user-defined functions
• Optimize conversion between Apache Spark and pandas DataFrames

Tools

In addition to Databricks notebooks, you can use the following Python developer tools:

• Jupyter with Databricks Connect
• PyCharm with Databricks Connect

Libraries

Databricks runtimes provide many libraries. To make third-party or locally-built Python libraries available to notebooks and jobs running on your Databricks clusters, you can install libraries following these instructions:

• Install Python libraries in a cluster
• Databricks library utilities
• Conda package management

Resources

• Migrate single node workloads to Databricks
• Knowledge Base