Skip to main content

Manage serverless base environments

Preview

This feature is in Public Preview.

This page explains how to create and manage serverless base environments across a workspace.

Permissions​

  • Only workspace admins can create and manage a workspace's base environments.
  • All workspace users have access to a workspace's base environments.
  • All workspace users can create custom serverless environment specifications.

How base environments work in Databricks​

In Databricks, a base environment is a shareable YAML specification that defines a serverless environment version and a set of additional Python dependencies for serverless notebooks. Workspace admins create and manage base environments so users can quickly start from a consistent, cached environment and optionally add their own libraries.

Create and export an environment specification​

The simplest way to create a valid YAML specification is to build the environment in the Environment side panel and then use the Export environment button to download the YAML file.

  1. Open a notebook and connect to serverless compute.
  2. Click the Environment Environment icon. button in the notebook's side panel.
  3. Select the Environment version. Databricks recommends using the latest serverless environment version supported by your workspace.
  4. In the Dependencies field, add whatever dependencies you would like the base environment to have. Click Add dependency after you enter each dependency. For more instructions on adding dependencies, see Add dependencies to the notebook.
  5. Click Apply on the bottom of the environment panel to ensure the specification is valid.
  6. Click the kebab menu icon Kebab menu icon. at the bottom of the environment panel then click Export environment.
  7. Give the YAML file a name and add to a Workspace folder or Unity Catalog volume.

Example environment specification​

The following example YAML is based on the MLflow projects environment specification. It defines a base environment with a few library dependencies:

YAML
environment_version: '4'
dependencies:
  - --index-url https://pypi.org/simple
  - -r "/Workspace/Shared/requirements.txt"
  - my-library==6.1
  - /Workspace/Shared/Path/To/simplejson-3.19.3-py3-none-any.whl
  - git+https://github.com/databricks/databricks-cli

Add a base environment to your workspace​

To add the environment specification as a base environment to the workspace:

  1. In the workspace, go to Settings.
  2. Under Workspace admin, select Compute.
  3. Next to Base environments for serverless compute, click Manage.
  4. Click Create new environment.
  5. Give your base environment a name. This is the name that users will see in the Base environment dropdown menu.
  6. Select the environment specification YAML file using the file picker. You can browse workspace files or Unity Catalog volumes.
  7. Click Create.

The base environment will start building. Check the Status column in the list of base environments. It will change to Ready to use when it's ready.

Set the workspace's default base environment​

By default, serverless notebooks in a workspace don't use a base environment. Workspace admins can select a base environment to apply to all new notebooks by default.

  1. In the workspace, go to Settings.
  2. Under Workspace admin, select Compute.
  3. Next to Base environments for serverless compute, click Manage.
  4. Click the star icon next to the base environment to set it as the default.

All new serverless notebooks will now default to the selected base environment.

Update a base environment​

You might want to edit the base environment file to update version numbers or add or remove dependencies.

In the list of base environments, click the YAML file path of the base environment you want to update. This opens up the file in a new tab. You can review or update the file contents there. Changes are saved automatically.

After you make an update to the YAML specification, you must refresh the base environment so notebooks and jobs pick up the latest configuration.

  1. Next to the base environment you want to refresh, click the kebab menu icon Kebab menu icon. then select Refresh.
  2. Click Confirm.

New sessions now use the updated base environment. Existing notebook sessions must be restarted to get the updates.

Limitations​

  • For jobs, only notebook tasks can use base environments.
  • Serverless environment version 1 is not supported. Use version 2 or higher.
  • Base environments are available to all workspace users.
  • Workspaces are limited to 10 base environments.
Last updated on