Databricks SDK for Python

Experimental

The Databricks SDK for Python is in an Experimental state. To provide feedback, ask questions, and report issues, use the Issues tab in the Databricks SDK for Python repository in GitHub.

During the Experimental period, Databricks is actively working on stabilizing the Databricks SDK for Python’s interfaces. API clients for all services are generated from specification files that are synchronized from the main platform. You are highly encouraged to install a specific version of the Databricks SDK for Python package and read the changelog where Databricks documents the changes. Databricks may have minor documented backward-incompatible changes, such as renaming the functions or some class names to bring more consistency.

In this article, you learn how to automate operations in Databricks accounts, workspaces, and related resources with the Databricks SDK for Python.

Before you begin

Before you begin to use the Databricks SDK for Python, your development machine must have:

Databricks authentication configured.
Python 3.8 or higher installed. (Python 3.7 is also supported but only through June 2023.) For automating Databricks compute resources, recommends that you have the major and minor versions of Python installed that match the one that is installed on your target Databricks compute resource. This article’s examples rely on automating clusters with Databricks Runtime 13.0, which has Python 3.10 installed. For the correct version, see Databricks runtime releases for your cluster’s Databricks Runtime version.
Databricks recommends that you create and activate a Python virtual environment for each Python code project that you use with the Databricks SDK for Python. Python virtual environments help to make sure that your code project is using compatible versions of Python and Python packages (in this case, the Databricks SDK for Python package). This article uses venv for Python virtual environments. To create a Python virtual environment with venv:
1. From your terminal set to the root directory of your Python code project, instruct venv to use Python 3.10 for the virtual environment, and then create the virtual environment’s supporting files in a hidden directory named .venv within the root directory of your Python code project, by running the following command:
```
# Linux and macOS
python3.10 -m venv ./.venv

# Windows
python3.10 -m venv .\.venv
```
2. Use venv to activate the virtual environment. See the venv documentation for the correct command to use, based on your operating system and terminal type. For example, on macOS running zsh:
```
source ./.venv/bin/activate
```
  You will know that your virtual environment is activated when the virtual environment’s name (for example, .venv) displays in parentheses just before your terminal prompt.
  
  To deactivate the virtual environment at any time, run the command deactivate.
  
  You will know that your virtual environment is deactivated when the virtual environment’s name no longer displays in parentheses just before your terminal prompt.

Get started with the Databricks SDK for Python

On your development machine with Databricks authentication configured, Python already installed, and your Python virtual environment already activated, use pip to install the databricks-sdk package from the Python Package Index (PyPI), as follows:
```
pip3 install databricks-sdk
```
To install a specific version of the databricks-sdk package (especially while the Databricks SDK for Python is in an Experimental state), see the package’s Release history. For example, to install version 0.1.6:
```
pip3 install databricks-sdk>=0.1.6
```
Tip

To upgrade an existing installation of the Databricks SDK for Python package to the latest version, run pip as follows:
```
pip3 install --upgrade databricks-sdk
```
To show the Databricks SDK for Python package’s current Version and other details, run pip as follows:
```
pip3 show databricks-sdk
```
In your Python virtual environment, create a Python code file that imports the Databricks SDK for Python. The following example, in a file named main.py with the following contents, simply lists all the clusters in your Databricks workspace:
```
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

for c in w.clusters.list():
  print(c.cluster_name)
```
Run your Python code file, assuming a file named main.py, by running the python command:
```
python3.10 main.py
```
Note

By not setting any arguments in the preceding call to w = WorkspaceClient(), the Databricks SDK for Python uses its default process for trying to perform Databricks authentication. To override this default behavior, see the following authentication section.

Authenticate the Databricks SDK for Python with your Databricks account or workspace

The Databricks SDK for Python implements the Databricks client unified authentication standard, a consolidated and consistent architectural and programmatic approach to authentication. This approach helps make setting up and automating authentication with Databricks more centralized and predictable. It enables you to configure Databricks authentication once and then use that configuration across multiple Databricks tools and SDKs without further authentication configuration changes. For more information, including more complete code examples in Python, see Databricks client unified authentication.

Some of the available coding patterns to initialize Databricks authentication with the Databricks SDK for Python include:

Use Databricks default authentication by doing one of the following:
- Create or identify a custom Databricks configuration profile with the required fields for the target Databricks authentication type. Then set the DATABRICKS_CONFIG_PROFILE environment variable to the name of the custom configuration profile.
- Set the required environment variables for the target Databricks authentication type.
```
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
# ...
```
Hard-coding the required fields is supported but not recommended, as it risks exposing sensitive information in your code, such as Databricks personal access tokens. The following example hard-codes Databricks host and access token values for Databricks token authentication:
```
from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
  host  = 'https://...',
  token = '...'
)
# ...
```

Use Databricks Utilities

You can call most of the dbutils.fs and dbutils.secrets command within Databricks Utilities by using dbutils within WorkspaceClient. This code example lists the paths of all of the objects in the DBFS root of the workspace.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
d = w.dbutils.fs.ls('/')

for f in d:
  print(f.path)

Code examples

The following code examples demonstrate how to use the Databricks SDK for Python to create and delete clusters, run jobs, and list account-level groups. These code examples use the Databricks SDK for Python’s default Databricks authentication process.

For additional code examples, see the examples folder in the Databricks SDK for Python repository in GitHub.

Create a cluster
Permanently delete a cluster
Run a job
List account-level groups

Create a cluster

This code example creates a cluster with the specified Databricks Runtime version and cluster node type. This cluster has one worker, and the cluster will automatically terminate after 15 minutes of idle time.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

print("Attempting to create cluster. Please wait...")

c = w.clusters.create(
  cluster_name             = 'my-cluster',
  spark_version            = '12.2.x-scala2.12',
  node_type_id             = 'i3.xlarge',
  auto_termination_minutes = 15,
  num_workers              = 1
)

print(f"The cluster is now ready at " \
      f"{w.config.host}#setting/clusters/{c.cluster_id}/configuration\n")

Permanently delete a cluster

This code example permanently deletes the cluster with the specified cluster ID from the workspace.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

c_id = input('ID of cluster to delete (for example, 1234-567890-ab123cd4): ')

w.clusters.permanent_delete(cluster_id = c_id)

Run a job

This code example creates a Databricks job that runs the specified notebook on the specified cluster. As the code runs, it gets the existing notebook’s path, the existing cluster ID, and related job settings from the user at the terminal.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import JobTaskSettings, NotebookTask, NotebookTaskSource

w = WorkspaceClient()

job_name            = input("Some short name for the job (for example, my-job): ")
description         = input("Some short description for the job (for example, My job): ")
existing_cluster_id = input("ID of the existing cluster in the workspace to run the job on (for example, 1234-567890-ab123cd4): ")
notebook_path       = input("Workspace path of the notebook to run (for example, /Users/someone@example.com/my-notebook): ")
task_key            = input("Some key to apply to the job's tasks (for example, my-key): ")

print("Attempting to run the job. Please wait...\n")

j = w.jobs.create(
  job_name = job_name,
  tasks = [
    JobTaskSettings(
      description = description,
      existing_cluster_id = existing_cluster_id,
      notebook_task = NotebookTask(
        base_parameters = [""],
        notebook_path = notebook_path,
        source = NotebookTaskSource("WORKSPACE")
      ),
      task_key = task_key
    )
  ]
)

print(f"View the job at {w.config.host}/#job/{j.job_id}\n")

List account-level groups

This code example lists the display names for all of the available groups within the Databricks account.

from databricks.sdk import AccountClient

a = AccountClient()

for g in a.groups.list():
  print(g.display_name)

Additional resources

For more information, see: