Use serverless SQL warehouses

This article explains how to use serverless SQL warehouses and how to enable them for your account.

With serverless compute, the compute layer exists in your Databricks account rather than your AWS account. This gives users in your account instant access to fully managed and elastic compute resources. Serverless compute is supported for use with Databricks SQL. Serverless compute does not affect how Databricks Runtime clusters work with notebooks and jobs.

Serverless SQL warehouses do not have public IP addresses. For more architectural information, see Serverless compute.

Requirements

Account requirements

  • Your Databricks account must be on the E2 version of the platform.

  • Your Databricks account must not be on a free trial.

  • If your account was created before March 28, 2022, it needs to accept the updated terms of use.

Workspace requirements

Serverless SQL warehouses support the compliance security profile in some regions

Preview

This feature is in Public Preview.

Support for serverless SQL warehouses with the compliance security profile varies by region. On the regions page, in the column for Serverless SQL warehouses, look for compliance security profile supported:

  • In those regions, serverless warehouses support workspaces that have the compliance security profile enabled. They have hardened images, encrypted inter-node communication, anti-virus monitors, file integrity monitors, and auto-restart for long-running serverless SQL warehouses.

  • Outside those regions, if the compliance security profile is enabled on a workspace, Databricks does not allow starting serverless SQL warehouses.

Important

Serverless SQL warehouse support for the compliance security profile will be incrementally rolled out to all customers in the two regions over several weeks. To prioritize the enrollment of your account for using the preview, contact your Databricks account team.

Also, the related feature enhanced security monitoring does not apply to serverless SQL warehouses.

Serverless quotas

Serverless quotas are a safety measure for serverless compute. Serverless quotas restrict how many serverless compute resources a customer can have at any given time. The quota is enforced at the regional level for all workspaces in your account. Quotas are enforced only for serverless SQL warehouses. See Serverless quotas.

Enable serverless compute in your account

Note

If your account was created after March 28, 2022, serverless compute is enabled by default for your workspaces.

If you are not an account admin, you cannot perform these steps. Contact an account admin if your workspace needs access to serverless compute.

  1. As an account admin, go to the feature enablement tab of the account console settings page.

  2. A banner at the top of the page prompts you to accept the additional terms. Once you read the terms, click Accept. If you do not see the banner asking you to accept the terms, this step has been completed already.

After you’ve accepted the terms, your account is enabled for serverless.

Update instance profile for serverless compute (optional)

If your workspace uses an instance profile created before June 24, 2022 for data access, you may need to update the instance profile’s trust relationship to enable serverless. See Update instance profiles for serverless. This doesn’t apply to workspaces enabled for Unity Catalog.

Create a serverless SQL warehouse

If your workspace is enabled for Databricks SQL Serverless, new SQL warehouses are serverless by default. The workspace must be in a supported region for serverless SQL warehouses and on the Premium plan or above. For other requirements, see Requirements.

You can also upgrade an existing pro or classic SQL warehouse to serverless.

If you use the SQL Warehouse API to create the warehouse, serverless is not the default. Databricks recommends always explicitly setting enable_serverless_compute to true and warehouse_type to pro.

Auto-restart of serverless SQL warehouses

A serverless SQL warehouse auto-restarts in the following conditions:

  • You attempt to run a query that uses a stopped warehouse.

  • A job assigned to a stopped warehouse is scheduled to run.

  • You connect to a stopped warehouse from a JDBC/ODBC interface.

  • You open a query in the SQL editor that is saved to a stopped warehouse.

  • A dashboard is opened that is saved with a dashboard-level warehouse assigned.

See Warehouse auto-restart.

Remove Hive metastore credentials to enable serverless

If the serverless option is unavailable when creating a new SQL warehouse, it might be because there are lingering Hive metastore credentials in your workspace preventing access to serverless warehouses. To remove legacy Hive metastore credentials:

  1. Click your username in the top bar of the workspace and select Admin Settings from the drop-down.

  2. Click the Compute tab.

  3. Click Manage next to SQL warehouses.

  4. In the Data Access Configuration field, locate and delete the Hive metastore credentials.

  5. Click Save changes.

You are now able to create serverless warehouses in your workspace.

Limitations

Serverless warehouses have the following limitations: