Libraries

To make third-party or custom code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Important

Default behavior for the library upload UI has changed. Legacy behavior always stored libraries in the DBFS root. All workspace users have the ability to modify data and files stored in the DBFS root.

The default location for library uploads is now workspace files. Databricks recommends uploading libraries to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.

This article focuses on performing library tasks in the workspace UI. You can also manage libraries using the CLI or the Libraries API.

Databricks includes many common libraries in Databricks Runtime. To see which libraries are included in Databricks Runtime, look at the System Environment subsection of the Databricks Runtime release notes for your Databricks Runtime version.

This article focuses on performing library tasks in the workspace UI. You can also manage libraries using the Libraries API.

Important

Databricks does not invoke Python atexit functions when your notebook or job completes processing. If you use a Python library that registers atexit handlers, you must ensure your code calls required functions before exiting.

Installing Python eggs is deprecated and not supported with Databricks Runtime 14.0 and above. Use Python wheels or install packages from PyPI instead.

Note

Unity Catalog has some limitations on library usage. For details, see Cluster libraries.

You can install libraries scoped to compute or notebooks.

  • Cluster libraries can be used by all notebooks running on a cluster. See Cluster libraries. You can install a cluster library directly from the following sources:

    • A public repository such as PyPI, Maven, or CRAN.

    • Workspace files.

    • Unity Catalog volumes.

    • A cloud object storage location.

    • Uploading library files from your local machine.

      • Note: Libraries installed directly from upload are stored as workspace files, by default.

  • Notebook-scoped libraries, available for Python and R, allow you to install libraries and create an environment scoped to a notebook session. These libraries do not affect other notebooks running on the same cluster. Notebook-scoped libraries do not persist and must be re-installed for each session. Use notebook-scoped libraries when you need a custom environment for a specific notebook.

Important

Workspace libraries have been deprecated. See (Legacy) Workspace libraries.

Python environment management

The following table provides an overview of options you can use to install Python libraries in Databricks.

Python package source

Notebook-scoped libraries with %pip

Cluster libraries

Job libraries with Jobs API

PyPI

Use %pip install. See example.

Select PyPI as the source.

Add a new pypi object to the job libraries and specify the package field.

Private PyPI mirror, such as Nexus or Artifactory

Use %pip install with the --index-url option. Secret management is available. See example.

Not supported.

Not supported.

VCS, such as GitHub, with raw source

Use %pip install and specify the repository URL as the package name. See example.

Select PyPI as the source and specify the repository URL as the package name.

Add a new pypi object to the job libraries and specify the repository URL as the package field.

Private VCS with raw source

Use %pip install and specify the repository URL with basic authentication as the package name. Secret management is available. See example.

Not supported.

Not supported.

File path

Use %pip install. See example.

Select File path/S3 as the source.

Add a new egg or whl object to the job libraries and specify the file path as the package field.

S3

Use %pip install together with a pre-signed URL. Paths with the S3 protocol s3:// are not supported.

Select File path/S3 as the source.

Add a new egg or whl object to the job libraries and specify the S3 path as the package field.

Python library precedence

You might encounter a situation where you need to override the version for a built-in library, or have a custom library that conflicts in name with another library installed on the cluster. When you run import <library>, the library with the high precedence is imported.

Important

Libraries stored in workspace files have different precedence depending on how they are added to the Python sys.path. Databricks Repos adds the current working directory to the path before all other libraries, while notebooks outside Repos add the current working directory after other libraries are installed. If you manually append workspace directories to your path, these always have the lowest precedence.

The following list orders precedence from highest to lowest. In this list, a lower number means higher precedence.

  1. Libraries in the current working directory (Repos only).

  2. Libraries in the Repo (Git folder) root directory (Repos only).

  3. Notebook-scoped libraries (%pip install in notebooks).

  4. Cluster libraries (using the UI, CLI, or API).

  5. Libraries included in Databricks Runtime.

    • Libraries installed with init scripts might resolve before or after built-in libraries, depending on how they are installed. Databricks does not recommend installing libraries with init scripts.

  6. Libraries in the current working directory (not in Repos).

  7. Workspace files appended to the sys.path.