Databricks for Python developers

This section provides a guide to developing notebooks and jobs in Databricks using the Python language.

Python APIs

PySpark API

PySpark is the Python API for Apache Spark. These links provide an introduction to and reference for PySpark.

• Introduction to DataFrames
• Introduction to Structured Streaming
• PySpark API reference

pandas API (Koalas)

pandas is a Python API that makes working with “relational” data easy and intuitive. Koalas implements the pandas DataFrame API for Apache Spark.

• Koalas
  • Install Koalas
  • Koalas example notebook
  • Resources

Visualizations

Databricks Python notebooks support various types of visualizations using the display function.

• Visualizations in Python

You can also use the following third-party libraries to create visualizations in Databricks Python notebooks.

• Bokeh
• Matplotlib
• Plotly

Interoperability

This section describes features that support interoperability between Python and SQL.

• User-defined functions - Python
• pandas user-defined functions
• Optimize conversion between Apache Spark and pandas DataFrames

Tools

In addition to Databricks notebooks, you can use the following Python developer tools:

• Jupyter with Databricks Connect
• PyCharm with Databricks Connect

Libraries

Databricks runtimes provide many libraries. There are several options to make third-party or locally-built Python libraries available to notebooks and jobs running on your Databricks clusters.

Make library available to all notebooks and jobs running on a cluster

• Install Python libraries in a cluster
• Conda package management

Make library available only to a specific notebook

• Conda and pip magic commands (only available with Databricks Runtime 7.0 ML or higher)
• Databricks library utilities (only available with Databricks Runtime, not with Databricks Runtime ML)

Resources

• Migrate single node workloads to Databricks
• Knowledge Base