Serverless vs. classic compute for pipelines
Every Lakeflow pipelines pipeline runs its updates on either serverless or classic compute. The compute type is a per-pipeline setting that you choose in the pipeline's settings. It is not automatic: a pipeline runs on serverless compute only when you enable the Serverless setting, and runs on classic compute otherwise. This page compares the two options so you can choose the right one for each pipeline.
Databricks recommends serverless compute for almost all pipelines. With serverless, Databricks manages the infrastructure for you. There are no clusters to size, configure, secure, or grant permissions for, and you get capabilities that classic compute cannot offer, such as incremental refresh and vertical autoscaling. With classic compute, that infrastructure is your responsibility to configure and maintain. Serverless supports nearly everything classic compute does. The exceptions are the legacy Hive metastore and some network configurations. For most pipelines, choosing classic compute means taking on manual work that serverless would otherwise handle.
Incremental refresh for materialized views is available only on serverless pipelines. Materialized views that run on classic compute are always fully recomputed. If incremental refresh is a requirement for your workload, use serverless compute. See Incremental refresh for materialized views.
Compare compute options
The following table compares serverless and classic compute across the capabilities that most often drive the choice:
Capability | Serverless | Classic |
|---|---|---|
Infrastructure management | Databricks manages all infrastructure. There is no cluster configuration. | You must configure clusters, including autoscaling, instance types, and cluster policies. |
Incremental refresh for materialized views | Supported. Materialized views are refreshed incrementally whenever it is cost efficient, to lower compute costs. See Incremental refresh for materialized views. | Not supported. Materialized views are always fully recomputed. |
Autoscaling | Enhanced autoscaling that scales horizontally (more executors) and vertically (larger executors). See Optimize Lakeflow pipeline cluster utilization with autoscaling. | Enhanced autoscaling that scales horizontally. You select the instance types. |
Stream pipelining | Enabled by default. Microbatches run concurrently to improve throughput and latency. See Configure a serverless pipeline. | Not available. |
Compute creation permission | Not required. All workspace users can run serverless pipelines by default. | Required. Users need unrestricted cluster creation permission or access to a compute policy. |
Compute policies and instance types | Managed by Databricks. You do not set instance types or a compute policy. | You manually apply a compute policy and select worker and driver instance types. |
Unity Catalog | Always uses Unity Catalog. | Can use Unity Catalog or the legacy Hive metastore. |
Cost attribution | Apply custom tags with a serverless usage policy. | Apply custom tags to the pipeline. You must manually join tag data to billing data. |
Update and maintenance clusters | Managed by Databricks. | You configure the update and maintenance clusters separately. |
Single-node compute | Not needed. Databricks sizes compute for the workload automatically. | You configure single-node compute for small or non-distributed workloads. |
Incremental refresh for materialized views is in Public Preview.
When to use serverless compute
Use serverless compute for any pipeline that does not hit one of the classic-only limitations below. In particular, serverless is the right choice when:
- You want Databricks to manage infrastructure for you, including vertical autoscaling and instance selection, instead of configuring and maintaining clusters yourself.
- You want incremental refresh for materialized views to reduce refresh costs.
- You want workspace users to run pipelines without cluster creation permissions.
Serverless pipelines require a workspace with Unity Catalog enabled and a serverless-enabled region. For requirements and setup, see Configure a serverless pipeline.
When to use classic compute
Serverless supports nearly every pipeline workload, so use classic compute only when serverless cannot run your pipeline at all:
- Your tables use the legacy Hive metastore instead of Unity Catalog. To migrate Hive metastore tables to Unity Catalog and enable serverless, see Upgrade Hive tables and views to Unity Catalog.
- Your pipeline requires private networking that serverless does not support.
- Your pipeline runs in a region where serverless is not available.
With classic compute, compute policies, instance types, single-node compute, and separate update and maintenance clusters are yours to configure and maintain, work that serverless does for you.
For setup and configuration options, see Configure classic compute for pipelines.
Set the compute type
You choose the compute type in the pipeline's Compute settings by enabling or disabling the Serverless setting. New pipelines use serverless by default. You can also convert an existing pipeline configured with Unity Catalog to serverless.
- To configure and convert serverless pipelines, see Configure a serverless pipeline.
- To configure classic compute, see Configure classic compute for pipelines.