Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines.
Enhanced Autoscaling improves on the Databricks cluster autoscaling functionality with the following features:
Enhanced Autoscaling implements optimization of streaming workloads, and adds enhancements to improve the performance of batch workloads. Enhanced Autoscaling optimizes costs by adding or removing machines as the workload changes.
Enhanced Autoscaling proactively shuts down under-utilized nodes while guaranteeing there are no failed tasks during shutdown. The existing cluster autoscaling feature scales down nodes only if the node is idle.
Enhanced Autoscaling is the default autoscaling mode when you create a new pipeline in the Delta Live Tables UI. You can enable Enhanced Autoscaling for existing pipelines by editing the pipeline settings in the UI. You can also enable Enhanced Autoscaling when you create or edit pipelines with the Delta Live Tables API.
To use Enhanced Autoscaling, do one of the following:
Set Cluster mode to Enhanced autoscaling when you create a pipeline or edit a pipeline in the Delta Live Tables UI.
autoscalesetting to the pipeline cluster configuration and set the
ENHANCED. See Configure your compute settings. The following example configures an Enhanced Autoscaling cluster with a minimum of 5 workers and a maximum of 10 workers.
max_workersmust be greater than or equal to
Enhanced Autoscaling is available for
updatesclusters only. The existing autoscaling feature is used for
autoscaleconfiguration has two modes:
LEGACY: Use cluster autoscaling.
ENHANCED: Use Enhanced Autoscaling.
The pipeline is automatically restarted after the autoscaling configuration changes if the pipeline is configured for continuous execution. After restart, expect a short period of increased latency. Following this brief period of increased latency, the cluster size should be updated based on your
autoscale configuration, and the pipeline latency returned to its previous latency characteristics.
You can use the event log in the Delta Live Tables user interface to monitor Enhanced Autoscaling metrics. Enhanced Autoscaling events have the
autoscale event type. The following are example events:
Cluster resize request started
Cluster resize request succeeded
Cluster resize request partially succeeded
Cluster resize request failed
You can also view Enhanced Autoscaling events by directly querying the event log:
To query the event log for backlog metrics, see Monitor data backlog by querying the event log.
To monitor cluster resizing requests and responses during Enhanced Autoscaling operations, see Monitor Enhanced Autoscaling events from the event log.