Enable serverless SQL warehouses

This article explains how to enable serverless SQL warehouses for your workspace.

Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Databricks architecture overview.

Requirements

Note

Your workspace can use AWS Glue as the workspace legacy metastore. See Use AWS Glue Data Catalog as a metastore (legacy).

Enable serverless compute in your account

Note

If your account was created after March 28, 2022, serverless compute is enabled by default for your workspaces.

  1. Go to the feature enablement tab of the account console settings page.

  2. A banner at the top of the page prompts you to accept the additional terms. Once you read the terms, click Accept. If you do not see the banner asking you to accept the terms, this step has been completed already.

After you’ve accepted the terms, your account is enabled for serverless.

Note

If your workspace has the compliance security profile enabled, you won’t be able to enable serverless compute yourself. You must reach out to your account team to enable it. See Serverless support with the compliance security profile.

Update instance profile for serverless compute (optional)

If your workspace uses an instance profile created before June 24, 2022 for data access, you may need to update the instance profile’s trust relationship to enable serverless. See Confirm or set up an AWS instance profile to use with your serverless SQL warehouses.

You do not need to complete this step if you are using Unity Catalog for data governance.

Serverless SQL warehouses support the compliance security profile in some regions

Preview

This feature is in Public Preview.

Support for serverless SQL warehouses with the compliance security profile varies by region. On the regions page, in the column for Serverless SQL warehouses, look for compliance security profile supported:

  • In those regions, workspaces that use the compliance security profile support serverless SQL warehouses. These workspaces have hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses. Reach out to your account team to get serverless SQL warehouses enabled in your workspace.

  • Outside those regions, if the compliance security profile is enabled on a workspace, Databricks does not allow starting serverless SQL warehouses.

The related feature enhanced security monitoring does not apply to serverless SQL warehouses.

Remove Hive metastore credentials to enable serverless

You cannot deploy serverless SQL warehouses if Hive metastore credentials are defined at the workspace level. To remove legacy Hive metastore credentials:

  1. Click your username in the top bar of the workspace and select Settings from the drop-down.

  2. Click the Compute tab.

  3. Click Manage next to SQL warehouses.

  4. In the Data Access Configuration field, locate and delete the Hive metastore credentials.

  5. Click Save changes.

Monitor usage with tags

Like any other compute resource in Databricks, you can apply tags to serverless SQL warehouses to monitor usage and cost. You can view usage in the account console or access downloadable logs. See Import a usage dashboard.

Limitations

Serverless warehouses have the following limitations:

  • Cluster policies, including spot instance policies are unsupported. See Create and manage compute policies.

  • Customer-managed VPCs are not applicable to compute resources for serverless SQL warehouses. See Configure a customer-managed VPC.

  • Serverless SQL warehouses use private connectivity between the Databricks control plane and the serverless compute plane in nearly all cases. Some legacy us-east-1 region workspaces use an alternative secure network connection.

  • Although the serverless compute plane does not use the secure cluster connectivity relay for the classic compute plane, serverless SQL warehouses do not have public IP addresses.

  • Serverless SQL warehouses do not use customer-managed keys for EBS storage encryption. See Serverless compute and customer-managed keys.