Use serverless SQL warehouses

This article explains how to use serverless SQL warehouses and how to enable them for your account.

With serverless compute, the compute layer exists in your Databricks account rather than your AWS account. This gives users in your account instant access to fully managed and elastic compute resources. Serverless compute is supported for use with Databricks SQL. Serverless compute does not affect how Databricks Runtime clusters work with notebooks and jobs.

Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Serverless compute.

Requirements

Account requirements

Your Databricks account must be on the E2 version of the platform.
Your Databricks account must not be on a free trial.
If your account was created before March 28, 2022, it needs to accept the updated terms of use.

Workspace requirements

Your Databricks workspace must be on the Premium or higher pricing tier.
Your workspace must be in a region that supports Databricks SQL Serverless.
Your workspace must not use an external Hive legacy metastore. However, the workspace can use AWS Glue as the workspace legacy metastore.
Your workspace must not use S3 access policies.

Serverless SQL warehouses support the compliance security profile in some regions

Preview

This feature is in Public Preview.

Support for serverless SQL warehouses with the compliance security profile varies by region.

Now in Public Preview, serverless SQL warehouses support the compliance security profile in some regions. On the regions page, in the column for Serverless SQL warehouses, look for compliance security profile supported:

In those regions, serverless warehouses support workspaces that have the compliance security profile enabled. They have hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses.
Outside those regions, if the compliance security profile is enabled on a workspace, Databricks does not allow starting serverless SQL warehouses.

Important

As part of this preview, the serverless SQL support for the compliance security profile will be incrementally rolled out to all customers in the us-east-1 region over several weeks. To prioritize the enrollment of your account for using the preview, contact your Databricks representative.

Also note that the related feature Enhanced Security Monitoring does not apply to serverless SQL warehouses.

Serverless quotas

Serverless quotas are a safety measure for serverless compute. Serverless quotas restrict how many serverless compute resources a customer can have at any given time. The quota is enforced at the regional level for all workspaces in your account. Quotas are enforced only for serverless SQL warehouses. See Serverless quotas.

Enable serverless compute in your account

Note

If your account was created after March 28, 2022, serverless compute is enabled by default for your workspaces.

If you are not an account admin, you cannot perform these steps. Contact an account admin if your workspace needs access to serverless compute.

As an account admin, go to the feature enablement tab of the account console settings page.
A banner at the top of the page prompts you to accept the additional terms. Once you read the terms, click Accept. If you do not see the banner asking you to accept the terms, this step has been completed already.

After you’ve accepted the terms, your account is enabled for serverless.

Update instance profile for serverless compute (optional)

If your workspace uses an instance profile created before June 24, 2022 for data access, you may need to update the instance profile’s trust relationship to enable serverless. See Update instance profiles for serverless. This doesn’t apply to workspaces enabled for Unity Catalog.

Create a serverless SQL warehouse

If your workspace is enabled for Databricks SQL Serverless, new SQL warehouses are serverless by default. The workspace must be in a supported region for serverless SQL warehouses and on the Premium pricing tier. For other requirements, see Requirements.

You can also upgrade an existing pro or classic SQL warehouse to serverless.

If you use the SQL Warehouse API to create the warehouse, serverless is not the default. Databricks recommends always explicitly setting enable_serverless_compute to true and warehouse_type to pro.

Auto-restart of serverless SQL warehouses

A serverless SQL warehouse auto-restarts in the following conditions:

You attempt to run a query that uses a stopped warehouse.
A job assigned to a stopped warehouse is scheduled to run.
You connect to a stopped warehouse from a JDBC/ODBC interface.
You open a query in the SQL editor that is saved to a stopped warehouse.
A dashboard is opened that is saved with a dashboard-level warehouse assigned.

See Warehouse auto-restart.

Limitations

Serverless warehouses have the following limitations:

Cluster policies, including spot instance policies are unsupported.
Customer-managed VPCs are not applicable to compute resources for serverless SQL warehouses.
Regardless of whether you have enabled AWS PrivateLink connectivity for the classic data plane, serverless SQL warehouses use private connectivity between the Databricks control plane and the serverless data plane in nearly all cases. Some legacy us-east1 region workspaces use an alternative secure network connection.
Although the serverless data plane does not use the secure cluster connectivity relay for the classic data plane, serverless SQL warehouses do not have public IP addresses.
Serverless SQL warehouses do not use customer-managed keys for EBS storage encryption, which is an optional part of the customer-managed keys for workspace storage feature configuration. Disks for serverless compute resources are short-lived and tied to the lifecycle of the serverless workload. For example, when serverless SQL warehouses are stopped or scaled down, the VMs and their storage are destroyed. See Serverless compute and customer-managed keys.