Learn the Databricks foundations
Learn how an account, workspaces, and Unity Catalog metastores relate before you set up your infrastructure. Every Databricks deployment uses these few objects. The decisions that you make about them, such as how many workspaces to create and in which regions, affect every later step.
Accounts, workspaces, and metastores
These three objects control who manages what, where workloads run, and how you govern data:
- Account: The top-level construct that you use to manage Databricks across your organization. In the account, you manage users, groups, and service principals, create workspaces, create Unity Catalog metastores, and manage usage and billing.
- Workspaces: The collaboration environments where users run workloads, such as data ingestion, interactive exploration, scheduled jobs, and machine learning training. An account can contain multiple workspaces, such as one for each environment.
- Unity Catalog metastores: The central governance system for data and AI assets, such as tables, volumes, and models. A metastore organizes data in a three-level namespace,
<catalog>.<schema>.<object>. Each region has one metastore, attached to the workspaces in that region, so that every workspace has the same view of the data and you manage access to it in one place. Databricks enables Unity Catalog automatically for workspaces created after November 8, 2023, so your account might already have a metastore in your region. See Get started with Unity Catalog.
Control plane and compute plane
Databricks has a control plane and a compute plane:
- Control plane: The backend services that Databricks manages, including the web application.
- Compute plane: Where compute processes your data. Serverless compute runs in a serverless compute plane that Databricks manages. Classic compute runs in your own cloud account, in the classic compute plane.
Because classic compute runs in your cloud account, you plan a cloud tenant layout before you create classic workspaces. See High-level architecture.
Troubleshooting
These five mistakes are common in new accounts and hard to undo after teams start building on the setup.
Notebooks use hardcoded credentials or environment variables
Route data access through Unity Catalog. A connection string or secret in a notebook bypasses the grants, lineage, and audit logs that Unity Catalog provides, and you can't revoke it centrally. Wiring data access at the cluster level, with external libraries and environment variables, also bypasses Unity Catalog. It creates dependencies that lineage doesn't show. Default to Unity Catalog even for quick experiments.
Account-admin rights are granted for workspace-only needs
Account admins manage the account: workspaces, identity, billing, and metastores. Workspace admins manage objects and settings inside one workspace. Don't give account-admin rights to someone who needs only workspace access. See Admin privileges in Unity Catalog.
Development, staging, and production share one workspace
Run development and production in the same workspace, and an untested job can break production pipelines. Separate development, staging, and production workspaces let you test code and workspace settings in development or staging before you apply them to production.
Each team has its own workspace
Splitting by team creates administrative overhead and makes cross-team collaboration harder. Add teams to a shared workspace as groups with the right permissions. Split further only when teams have different platform needs or a hard compliance rule requires it. See Phase 2: Design workspace strategy.
A workspace is assumed to isolate data
A workspace doesn't isolate data by itself. Isolation comes from Unity Catalog grants, and from workspace bindings that restrict which workspaces can access a catalog. Without a binding, a workspace can reach any catalog in its metastore that the user has privileges on. See Workspace-catalog binding.
Key terms
Use these definitions for the roles and planes that the rest of the Starter Journey refers to:
Term | Definition |
|---|---|
Account admin | A role with control over the whole account: workspaces, identity, billing, and metastores |
Workspace admin | A role with admin control over one workspace |
Metastore admin | An optional role that governs one Unity Catalog metastore, including ownership and privileges on objects in it |
Control plane | The backend services that Databricks manages, including the web application |
Compute plane | The plane where compute processes your data: your cloud account for classic compute, and a plane that Databricks manages for serverless compute |
Key takeaways
Three rules from this page apply to every later section:
- The account console manages the account, and workspaces do the data work.
- Plan separate development, staging, and production workspaces, and add teams to them as groups.
- Unity Catalog is where you govern data access, so route every data and AI interaction through it.
Learn more
The guides in this table go deeper on each topic:
Topic | Description | Guide |
|---|---|---|
Databricks concepts | Accounts, workspaces, compute, data objects, and the other concepts that you use every day | |
The workspace UI | How to find the tools in the workspace sidebar, search, and the workspace browser | |
Unity Catalog | How metastores, catalogs, schemas, and privileges govern your data |
Next steps
-
- Plan your cloud tenant layout
- Decide whether your workspaces share one cloud tenant or use a separate tenant for each environment
-
- High-level architecture
- Learn how the control plane, the compute plane, and your cloud account fit together