Skip to main content

Compute selection recommendations

Choose a compute type for your notebooks, jobs, pipelines, or SQL workloads in Databricks. The compute resources you can choose or create depend on your permissions.

For guidance on configuring classic compute resources, see Classic compute configuration best practices.

Choose compute for your interactive notebooks​

Compute type

When to use

Serverless compute

Generally recommended. Offers faster start-up, automatic scaling, and lower cost.

Serverless SQL warehouse

For SQL-based analytics and reporting. Optimized for SQL query performance.

Classic all-purpose compute

When you need RDD APIs or R language support.

Compute type

When to use

Serverless compute

Generally recommended. Offers faster start-up, automatic scaling, and lower cost.

Serverless SQL warehouse

For SQL-based analytics and reporting. Optimized for SQL query performance.

Classic all-purpose compute

When you need RDD APIs or R language support.

For more on notebook compute, see Notebook compute resources.

Choose compute for your jobs​

When configuring a job, select a compute resource for each task.

Compute type

When to use

Serverless compute

Recommended for most automated workloads. Databricks manages the infrastructure and scaling. Choose a performance mode based on the workload's completion and data freshness requirements.

SQL warehouse

For SQL tasks. Use a warehouse type that fits your latency and cost needs.

Classic jobs compute

For non-SQL jobs that require custom cluster settings not available in serverless.

Classic all-purpose compute

Generally avoid. Not optimized for automated workloads.

Compute type

When to use

Serverless compute

Recommended for most automated workloads. Databricks manages the infrastructure and scaling. Choose a performance mode based on the workload's completion and data freshness requirements.

SQL warehouse

For SQL tasks. Use a warehouse type that fits your latency and cost needs.

Classic jobs compute

For non-SQL jobs that require custom cluster settings not available in serverless.

Classic all-purpose compute

Generally avoid. Not optimized for automated workloads.

To view a table of recommended compute types for each task type, see Recommended compute for each task.

For scheduled batch jobs on serverless compute, Databricks recommends evaluating standard mode for lower cost when representative runs meet your completion and data freshness requirements. Use performance-optimized mode for fast feedback during development or when standard mode cannot meet your timing requirements. New serverless jobs default to performance-optimized mode, so select standard mode explicitly. See Switch a job to standard mode.

For automated work on classic all-purpose compute, review serverless compatibility and create a job that uses serverless compute.

Choose compute for pipelines​

When configuring a pipeline, either use serverless compute or a classic pipeline.

Compute type

When to use

Serverless compute

Recommended for most automated workloads. Databricks manages the infrastructure and scaling. Choose a performance mode based on the workload's completion and data freshness requirements.

Classic pipeline compute

When you need a feature not supported in serverless or when you are working with the legacy Hive metastore.

Compute type

When to use

Serverless compute

Recommended for most automated workloads. Databricks manages the infrastructure and scaling. Choose a performance mode based on the workload's completion and data freshness requirements.

Classic pipeline compute

When you need a feature not supported in serverless or when you are working with the legacy Hive metastore.

Choose a SQL warehouse type for your SQL workloads​

There are three SQL warehouse types you can choose from:

Warehouse type

When to use

Serverless SQL warehouse

Recommended for BI, ETL, and exploratory analysis. Startup typically 2–6 seconds and scales rapidly.

Pro SQL warehouse

Use when serverless is not available or you require custom networking (for example, federation or hybrid).

Classic SQL warehouse

Entry-level performance. Use for basic interactive exploration when serverless or pro are not options.

Warehouse type

When to use

Serverless SQL warehouse

Recommended for BI, ETL, and exploratory analysis. Startup typically 2–6 seconds and scales rapidly.

Pro SQL warehouse

Use when serverless is not available or you require custom networking (for example, federation or hybrid).

Classic SQL warehouse

Entry-level performance. Use for basic interactive exploration when serverless or pro are not options.

For more considerations on choosing a SQL warehouse type, see SQL warehouse types.