Manage storage configurations using the account console (E2)

This article describes how to:

Note

This article describes the process for accounts on the E2 version of the Databricks platform, using the account console. To learn how to create storage configurations using the Account API, see Create a new workspace using the Account API. For other account types, see Configure AWS storage (Legacy). All new Databricks accounts and most existing accounts are now E2. If you are unsure which account type you have, contact your Databricks representative.

The bucket that you include in your storage configuration is referred to as your workspace’s root storage. Do not use your root storage to store production customer data. Instead, create additional S3 buckets or other data sources for production data and optionally create DBFS mount points for them.

Define a storage configuration and generate a bucket policy

Note

These instructions show you how to create the storage configuration from the Account Settings page in the account console before you create a new workspace. You can also create the storage configuration in a similar way as part of the flow of creating a new workspace. See Create and manage workspaces using the account console.

  1. Go to the account console, click Account Settings, and click Storage configurations.
  2. Click Add Storage Configuration.
  3. In the Storage Configuration Name field, enter a human-readable name for your new storage configuration.
  4. In the Bucket Name field, enter the exact name of the S3 bucket you will create.
  5. Click Generate Policy and copy the policy that is generated. You will add this to your S3 bucket configuration in AWS in the next task.
  6. Click Add.

Create the S3 bucket

  1. Log into your AWS Console as a user with administrator privileges and go to the S3 service.

  2. Create an S3 bucket, using the name that you entered in the Databricks storage configuration.

    See Create a Bucket in the AWS documentation.

    Important

    • The S3 bucket must be in the same AWS region as the Databricks workspace deployment.
    • Databricks recommends as a best practice that you use an S3 bucket that is dedicated to Databricks, unshared with other resources or services.
  3. Click the Permissions tab.

  4. Click the Bucket Policy button.

    Bucket policy button
  5. Paste the bucket policy that you generated and copied from the Add Storage Configuration dialog in Databricks.

  6. Save the bucket.

View storage configurations

  1. Go to the account console, click Account Settings, and click Storage configurations.

    All storage configurations are listed, with Bucket Name and Created date displayed for each.

  2. Click the storage configuration name to view more details.

Delete a storage configuration

Storage configurations cannot be edited after creation. If the configuration has incorrect data or if you no longer need it, delete the storage configuration:

  1. Go to the account console, click Account Settings, and click Storage configurations.

  2. On the storage configuration row, click the Actions menu icon, and select Delete.

    You can also click the storage configuration name and click Delete on the pop-up dialog.

  3. In the confirmation dialog, click Confirm Delete.

Encrypt your root S3 bucket using customer-managed keys (optional)

Preview

This feature is in Public Preview.

You can encrypt your root S3 bucket using customer-managed keys, which requires using the Account API. You can either add an encryption key when you create a new workspace using the Account API or add the key later. For more information, see Customer-managed keys for workspace storage.