Databricks architecture overview

This article provides a high-level overview of Databricks architecture, including its enterprise architecture, in combination with AWS.

High-level architecture

Databricks operates out of a control plane and a compute plane.

• The control plane includes the backend services that Databricks manages in your Databricks account. The web application is in the control plane.

• The compute plane is where your data is processed. There are two types of compute planes depending on the compute that you are using.

  • For serverless compute, the serverless compute resources run in a serverless compute plane in your Databricks account.

  • For classic Databricks compute, the compute resources are in your AWS account in what is called the classic compute plane. This refers to the network in your AWS account and its resources.

The following diagram describes the overall Databricks architecture.

Serverless compute plane

In the serverless compute plane, Databricks compute resources run in a compute layer within your Databricks account. Databricks creates a serverless compute plane in the same AWS region as your workspace’s classic compute plane.

To protect customer data within the serverless compute plane, serverless compute runs within a network boundary for the workspace, with various layers of security to isolate different Databricks customer workspaces and additional network controls between clusters of the same customer.

To learn more about networking in the serverless compute plane, Serverless compute plane networking.

Classic compute plane

In the classic compute plane, Databricks compute resources run in your AWS account. New compute resources are created within each workspace’s virtual network in the customer’s AWS account.

A classic compute plane has natural isolation because it runs in each customer’s own AWS account. To learn more about networking in the classic compute plane, see Classic compute plane networking.

For regional support, see Databricks clouds and regions.