Configure a serverless pipeline

This article describes configurations for serverless Lakeflow Declarative Pipelines.

Databricks recommends developing new pipelines using serverless. Some workloads might require configuring classic compute or working with the legacy Hive metastore. See Configure classic compute for Lakeflow Declarative Pipelines and Use Lakeflow Declarative Pipelines with legacy Hive metastore.

note

• Serverless pipelines always use Unity Catalog. See Use Unity Catalog with your Lakeflow Declarative Pipelines.
• For serverless compute limitations, see Serverless compute limitations.
• You cannot manually add compute settings in a clusters object in the JSON configuration for a serverless pipeline. Attempting to do so results in an error.

Requirements

• Your workspace must have Unity Catalog enabled to use serverless pipelines.

• You must have accepted the serverless terms of use.
• Your workspace must be in a serverless-enabled region.

Recommended configuration for serverless pipelines

important

Cluster creation permission is not required to configure serverless pipelines. By default, all workspace users can use serverless pipelines.

Serverless pipelines remove most configuration options, as Databricks manages all infrastructure. When you create a new pipeline, the default is to use serverless. To learn how to configure a serverless pipeline, see Configure Lakeflow Declarative Pipelines.

You can also convert existing pipelines configured with Unity Catalog to use serverless. See Convert an existing pipeline to use serverless.

Other configuration considerations

The following configuration options are also available for serverless pipelines:

• You might choose to use the Continuous pipeline mode when running pipelines in production. See Triggered vs. continuous pipeline mode.
• Add Notifications for email updates based on success or failure conditions. See Add email notifications for pipeline events.
• Use the Configuration field to set key-value pairs for the pipeline. These configurations serve two purposes:
  • Set arbitrary parameters you can reference in your source code. See Use parameters with Lakeflow Declarative Pipelines.
  • Configure pipeline settings and Spark configurations. See Lakeflow Declarative Pipelines properties reference.
• Use the Preview channel to test your pipeline against pending Lakeflow Declarative Pipelines runtime changes and trial new features.

Serverless budget policy

Preview

This feature is in Public Preview.

Serverless budget policies allow your organization to apply custom tags on serverless usage for granular billing attribution. After you select the Serverless checkbox, the Budget policy setting appears where you can select the policy you want to apply to the pipeline. The tags are inherited from the serverless budget policy and can only be edited by workspace admins.

note

After you've been assigned a serverless budget policy, your existing pipelines are not automatically tagged with your policy. You must manually update existing pipelines if you want to attach a policy to them.

For more on serverless budget policies, see Attribute usage with serverless budget policies.

Select a performance mode

For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard performance mode. Standard performance mode is designed to reduce costs for workloads where a slightly higher launch latency is acceptable. Serverless workloads using standard performance mode typically start within four to six minutes after being triggered, depending on compute availability and optimized scheduling.

When Performance optimized is enabled, your pipeline is optimized for performance, resulting in faster startup and execution for time-sensitive workloads.

Both modes use the same SKU, but standard performance mode consumes fewer DBUs, reflecting lower compute usage.

note

To use standard performance mode in continuous pipelines, reach out to your Databricks account team.

Serverless pipeline features

In addition to simplifying configuration, serverless pipelines have the following features:

• Stream pipelining: To improve utilization, throughput, and latency for streaming data workloads such as data ingestion, microbatches are pipelined. In other words, instead of running microbatches sequentially like standard Spark Structured Streaming, serverless Lakeflow Declarative Pipelines runs microbatches concurrently, improving compute resource utilization. Stream pipelining is enabled by default in serverless Lakeflow Declarative Pipelines.
• Vertical autoscaling: serverless Lakeflow Declarative Pipelines adds to the horizontal autoscaling provided by Databricks enhanced autoscaling by automatically allocating the most cost-efficient instance types that can run your pipeline without failing because of out-of-memory errors. See What is vertical autoscaling?

Convert an existing pipeline to use serverless

You can convert existing pipelines configured with Unity Catalog to serverless pipelines. Complete the following steps:

1. In your Databricks workspace's sidebar, click Jobs & Pipelines.
2. Click the Name of the pipeline.
3. Click Settings.
4. In the right sidebar, under Compute, click .
5. Check the box next to Serverless.
6. Click Save.

important

When you enable serverless, any compute settings you have configured for a pipeline are removed. If you switch a pipeline back to non-serverless updates, you must reconfigure the desired compute settings to the pipeline configuration.

How can I find the DBU usage of a serverless pipeline?

You can find the DBU usage of serverless Lakeflow Declarative Pipelines by querying the billable usage table, part of the Databricks system tables. See What is the DBU consumption of a serverless pipeline?.