Install libraries from a volume

Preview

This feature is in Public Preview.

This article walks you through the steps required to upload libraries to volumes and install them onto clusters in Databricks. For more information about volumes, see Create volumes. For information about working with Unity Catalog, including controlling access and creating objects, see What is Unity Catalog?.

In Databricks Runtime 13.2 and above, Python .whl libraries are supported.

In Databricks Runtime 13.3 and above, JAR libraries are supported in volumes. JARs cannot be installed at the notebook level. You must add the paths for JAR libraries to the allowlist to install them on compute with shared access mode. See Allowlist libraries and init scripts on shared compute.

Load libraries to a volume

To load a library to a volume:

  1. Click Catalog icon Catalog in the left sidebar.

  2. In the Catalog Explorer tree, navigate to the volume.

  3. Click Upload to this volume upload to volume button.

  4. The Upload files to volume dialog appears. Drag and drop or browse to the file(s) you want to upload, and click Upload.

Install libraries from a volume onto a cluster

You can only install libraries from a volume onto a shared-access cluster. See Create a cluster for information about cluster access modes.

When you install a library onto a cluster, all notebooks running on that cluster have access to the library.

To install a library from a volume onto a cluster:

  1. Click compute icon Compute in the left sidebar.

  2. Click the name of the cluster in the cluster list.

  3. Click the Libraries tab.

  4. Click Install new. The Install library dialog appears.

  5. For Library Source, select File path/S3, and for Library Type, select Python Whl or JAR.

  6. In the File Path field, enter the library’s volume location, such as the following: /Volumes/<catalog>/<schema>/<volume>/<path-to-library>/<library-name>.whl.

  7. Click Install.

Install libraries from a volume to a notebook

You can install Python libraries directly to a notebook to create custom Python environments that are specific to the notebook. For example, you can use a specific version of a library in a notebook, without affecting other users on the cluster who may need a different version of the same library. For more information, see notebook-scoped libraries.

When you install a library to a notebook, only the current notebook and any jobs associated with that notebook have access to that library. Other notebooks attached to the same cluster are not affected.

The following code shows how to install a Python wheel file from a volume to a notebook as a notebook-scoped library.

%pip install /Volumes/<catalog>/<schema>/<volume>/<path-to-library>/mypackage-0.0.1-py3-none-any.whl