This article explains how to use serverless SQL warehouses and how to enable them for your account.
With serverless compute, the compute layer exists in your Databricks account rather than your AWS account. This gives users in your account instant access to fully managed and elastic compute resources. Serverless compute is supported for use with Databricks SQL. Serverless compute does not affect how Databricks Runtime clusters work with notebooks and jobs.
Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Serverless compute.
Your Databricks account must be on the E2 version of the platform.
Your Databricks account must not be on a free trial.
Your Databricks workspace must be on the Premium plan or above.
Your workspace must be in a region that supports Databricks SQL Serverless.
Your workspace must not use an external Hive legacy metastore. However, the workspace can use AWS Glue as the workspace legacy metastore.
You must remove Hive metastore credentials from your workspace to use serverless. See Remove Hive metastore credentials to enable serverless.
Your workspace must not use S3 access policies.
This feature is in Public Preview.
Support for serverless SQL warehouses with the compliance security profile varies by region. On the regions page, in the column for Serverless SQL warehouses, look for compliance security profile supported:
In those regions, serverless warehouses support workspaces that have the compliance security profile enabled. They have hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses.
Outside those regions, if the compliance security profile is enabled on a workspace, Databricks does not allow starting serverless SQL warehouses.
Serverless SQL warehouse support for the compliance security profile will be incrementally rolled out to all customers in the two regions over several weeks. To prioritize the enrollment of your account for using the preview, contact your Databricks account team.
Also, the related feature enhanced security monitoring does not apply to serverless SQL warehouses.
Serverless quotas are a safety measure for serverless compute. Serverless quotas restrict how many serverless compute resources a customer can have at any given time. The quota is enforced at the regional level for all workspaces in your account. Quotas are enforced only for serverless SQL warehouses. See Serverless quotas.
If your account was created after March 28, 2022, serverless compute is enabled by default for your workspaces.
If you are not an account admin, you cannot perform these steps. Contact an account admin if your workspace needs access to serverless compute.
As an account admin, go to the feature enablement tab of the account console settings page.
A banner at the top of the page prompts you to accept the additional terms. Once you read the terms, click Accept. If you do not see the banner asking you to accept the terms, this step has been completed already.
After you’ve accepted the terms, your account is enabled for serverless.
If your workspace uses an instance profile created before June 24, 2022 for data access, you may need to update the instance profile’s trust relationship to enable serverless. See Update instance profiles for serverless. This doesn’t apply to workspaces enabled for Unity Catalog.
If your workspace is enabled for Databricks SQL Serverless, new SQL warehouses are serverless by default. The workspace must be in a supported region for serverless SQL warehouses and on the Premium plan or above. For other requirements, see Requirements.
If you use the SQL Warehouse API to create the warehouse, serverless is not the default. Databricks recommends always explicitly setting
A serverless SQL warehouse auto-restarts in the following conditions:
You attempt to run a query that uses a stopped warehouse.
A job assigned to a stopped warehouse is scheduled to run.
You connect to a stopped warehouse from a JDBC/ODBC interface.
You open a query in the SQL editor that is saved to a stopped warehouse.
A dashboard is opened that is saved with a dashboard-level warehouse assigned.
If the serverless option is unavailable when creating a new SQL warehouse, it might be because there are lingering Hive metastore credentials in your workspace preventing access to serverless warehouses. To remove legacy Hive metastore credentials:
Click your username in the top bar of the workspace and select Admin Settings from the drop-down.
Click the Compute tab.
Click Manage next to SQL warehouses.
In the Data Access Configuration field, locate and delete the Hive metastore credentials.
Click Save changes.
You are now able to create serverless warehouses in your workspace.
Serverless warehouses have the following limitations:
Cluster policies, including spot instance policies are unsupported.
Customer-managed VPCs are not applicable to compute resources for serverless SQL warehouses.
Regardless of whether you have enabled AWS PrivateLink connectivity for the classic compute plane, serverless SQL warehouses use private connectivity between the Databricks control plane and the serverless compute plane in nearly all cases. Some legacy
us-east1region workspaces use an alternative secure network connection.
Although the serverless compute plane does not use the secure cluster connectivity relay for the classic compute plane, serverless SQL warehouses do not have public IP addresses.
Serverless SQL warehouses do not use customer-managed keys for EBS storage encryption, which is an optional part of the customer-managed keys for workspace storage feature configuration. Disks for serverless compute resources are short-lived and tied to the lifecycle of the serverless workload. For example, when serverless SQL warehouses are stopped or scaled down, the VMs and their storage are destroyed. See Serverless compute and customer-managed keys.