Skip to main content

Configure a serverless pipeline

This article describes configurations for serverless Lakeflow Declarative Pipelines.

Databricks recommends developing new pipelines using serverless. Some workloads might require configuring classic compute or working with the legacy Hive metastore. See Configure classic compute for Lakeflow Declarative Pipelines and Use Lakeflow Declarative Pipelines with legacy Hive metastore.

note

Requirements​

  • Your workspace must have Unity Catalog enabled to use serverless pipelines.

Recommended configuration for serverless pipelines​

important

Cluster creation permission is not required to configure serverless pipelines. By default, all workspace users can use serverless pipelines.

Serverless pipelines remove most configuration options, as Databricks manages all infrastructure. To configure a serverless pipeline, do the following:

  1. In your workspace, click Workflows icon. Jobs & Pipelines in the sidebar.
  2. Under New, click ETL Pipeline.
  3. Provide a unique Pipeline name.
  4. Check the box next to Serverless.
  5. (Optional) Use the File Picker Icon file picker to configure notebooks and workspace files as Source code.
    • If you don't add any source code, a new notebook is created for the pipeline. The notebook is created in a new directory in your user directory, and a link to access this notebook is shown in the Source code field in the Pipeline details pane after you've created the pipeline.
      • A link to access this notebook is present under the Source code field in the Pipeline details panel once you've created your pipeline.
    • Use the Add source code button to add additional source code assets.
  6. Select a Catalog to publish data.
  7. Select a Schema in the catalog. All streaming tables and materialized views defined in the pipeline are created in this schema.
  8. Click Create.

These recommended configurations create a new pipeline configured to run in Triggered mode and the Current channel. This configuration is recommended for many use cases, including development and testing, and is well-suited to production workloads that should run on a schedule. For details on scheduling pipelines, see Pipeline task for jobs.

You can also convert existing pipelines configured with Unity Catalog to use serverless. See Convert an existing pipeline to use serverless.

Other configuration considerations​

The following configuration options are also available for serverless pipelines:

Serverless budget policy​

Preview

This feature is in Public Preview.

Serverless budget policies allow your organization to apply custom tags on serverless usage for granular billing attribution. After you select the Serverless checkbox, the Budget policy setting appears where you can select the policy you want to apply to the pipeline. The tags are inherited from the serverless budget policy and can only be edited by workspace admins.

note

After you've been assigned a serverless budget policy, your existing pipelines are not automatically tagged with your policy. You must manually update existing pipelines if you want to attach a policy to them.

For more on serverless budget policies, see Attribute usage with serverless budget policies.

Select a performance mode​

For triggered pipelines, you can select the serverless compute performance mode using the Performance optimized setting in the pipeline scheduler. When this setting is disabled, the pipeline uses standard performance mode. Standard performance mode is designed to reduce costs for workloads where a slightly higher launch latency is acceptable. Serverless workloads using standard performance mode typically start within four to six minutes after being triggered, depending on compute availability and optimized scheduling.

When Performance optimized is enabled, your pipeline is optimized for performance, resulting in faster startup and execution for time-sensitive workloads.

Serverless pipeline features​

In addition to simplifying configuration, serverless pipelines have the following features:

  • Stream pipelining: To improve utilization, throughput, and latency for streaming data workloads such as data ingestion, microbatches are pipelined. In other words, instead of running microbatches sequentially like standard Spark Structured Streaming, serverless Lakeflow Declarative Pipelines runs microbatches concurrently, improving compute resource utilization. Stream pipelining is enabled by default in serverless Lakeflow Declarative Pipelines.
  • Vertical autoscaling: serverless Lakeflow Declarative Pipelines adds to the horizontal autoscaling provided by Databricks enhanced autoscaling by automatically allocating the most cost-efficient instance types that can run your pipeline without failing because of out-of-memory errors. See What is vertical autoscaling?

Convert an existing pipeline to use serverless​

You can convert existing pipelines configured with Unity Catalog to serverless pipelines. Complete the following steps:

  1. In your Databricks workspace's sidebar, click Jobs & Pipelines.
  2. Click the Name of the pipeline.
  3. Click Pipelines in the sidebar.
  4. Click the name of the desired pipeline in the list.
  5. Click Settings.
  6. Check the box next to Serverless.
  7. Click Save and start.
important

When you enable serverless, any compute settings you have configured for a pipeline are removed. If you switch a pipeline back to non-serverless updates, you must reconfigure the desired compute settings to the pipeline configuration.

How can I find the DBU usage of a serverless pipeline?​

You can find the DBU usage of serverless Lakeflow Declarative Pipelines by querying the billable usage table, part of the Databricks system tables. See What is the DBU consumption of a serverless pipeline?.

Last updated on