Skip to main content

Configure default Python package repositories

Workspace admins can configure private or authenticated package repositories within workspaces as the default pip configuration for serverless notebooks, serverless jobs, and classic compute.

If a workspace is configured with a default Python package repo, users in the workspace will be able to install packages from internal Python repositories without explicitly defining index-url or extra-index-url values. However, if these values are specified in code or in a notebook, they take precedence over the workspace defaults.

This configuration leverages Databricks secrets to securely store and manage repository URLs and credentials. Workspace admins can configure the setup using the workspace admin settings page or using a predefined secret scope and the Databricks CLI secrets commands or the REST API.

Set up default dependencies for a workspace​

Workspace admins can add or remove the default Python package repositories using the workspace admin settings page.

  1. As a workspace administrator, log in to the Databricks workspace.
  2. Click your username in the top bar of the Databricks workspace and select Settings.
  3. Click on the Compute tab.
  4. Next to Default Package Repositories, click Manage.
  5. (Optional) Add or remove an index URL, extra index URLs, or a custom SSL certificate.
  6. Click Save to save the changes.
note

Modifications or deletions to secrets are applied after reattaching the compute resource to notebooks or rerunning the jobs.

Setup using the secrets CLI or REST API​

To configure default Python package repositories using the CLI or REST API, create a predefined secret scope and configure access permissions, then add the package repository secrets.

Predefined secret scope name​

Workspace administrators can set default pip index URLs or extra index URLs along with authentication tokens and secrets in a designated secret scope under predefined keys:

  • Secret scope name: databricks-package-management
  • Secret key for index-url: pip-index-url
  • Secret key for extra-index-urls: pip-extra-index-urls
  • Secret key for SSL certification content: pip-cert

Create the secret scope​

A secret scope can be created using the Databricks CLI secrets commands or the REST API. After creating the secret scope, configure access control lists to grant all workspace users read access. This ensures that the repository remains secure and cannot be altered by individual users. The secret scope must use the predefined secret scope name databricks-package-management.

Bash
databricks secrets create-scope databricks-package-management
databricks secrets put-acl databricks-package-management admins MANAGE
databricks secrets put-acl databricks-package-management users READ

Add Python package repository secrets​

Add the Python package repository details using the predefined secret key names, with all three fields being optional.

Bash
# Add index URL.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-index-url", "string_value":"<index-url-value>"}'

# Add extra index URLs. If you have multiple extra index URLs, separate them using white space.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-extra-index-urls", "string_value":"<extra-index-url-1 extra-index-url-2>"}'

# Add cert content. If you want to pip configure a custom SSL certificate, put the cert file content here.
databricks secrets put-secret --json '{"scope": "databricks-package-management", "key": "pip-cert", "string_value":"<cert-content>"}'

Modify or delete private PyPI repository secrets​

To modify PyPI repository secrets, use the put-secret command. To delete PyPI repository secrets, use delete-secret as shown below:

Bash
# delete secret
databricks secrets delete-secret databricks-package-management pip-index-url

databricks secrets delete-secret databricks-package-management pip-extra-index-urls

databricks secrets delete-secret databricks-package-management pip-cert

# delete scope
databricks secrets delete-scope databricks-package-management
Last updated on