Skip to main content

Plan your data governance strategy

Decide your catalog, schema, and grant structure before any team builds a pipeline. This decision determines whether teams can find data, trust it, and share it safely. A poorly planned structure leads to permission creep, duplicated tables, and unclear ownership. Every query, pipeline, and dashboard references tables by their full name, so moving tables later requires rewriting those references.

Unity Catalog organizes data and AI assets in a three-level namespace, catalog.schema.object, and grants access through privileges that child objects inherit from their parents. Your governance strategy maps the hierarchy to your environments, teams, and projects, so that each team can find the data that it needs, owners are clear, and you can audit access.

This section covers the namespace pattern for your organization's size, then object ownership and row filters and column masks for sensitive data.

Governance principles​

The patterns in this section apply the following Unity Catalog best practices:

  • Catalogs isolate data: Catalogs are the primary unit of data isolation in Unity Catalog. Create catalogs that correspond to environments, business units, or both, and bind each catalog to the workspaces that should access it. See Workspace-catalog binding.
  • Schemas organize projects: Within each catalog, create schemas for each project. The patterns in this section use one schema for each layer of the medallion architecture: bronze, silver, and gold. See What is the medallion lakehouse architecture?.
  • Groups receive access: Grant privileges and assign ownership to groups, not to individual users, so that you change access by changing group membership. Grant at the schema level, so that new tables in the schema inherit the grants.
  • Service principals write to production: Run production jobs and pipelines as service principals, and reserve write access to production tables for them, so that no one overwrites production data by accident.

The following example separates development, staging, and production workspaces, catalogs, and storage. User groups have read and write access in development and staging. In production, user groups have read access, and jobs that run as service principals have read and write access.

Development, staging, and production catalogs share one UC metastore. Each catalog is bound to its matching workspace, whose workloads access separate cloud object storage. User groups can read and write in development and staging. Production writes are reserved for jobs that run as service principals.

The lines between catalogs and workspaces show workspace bindings, which restrict the workspaces that can access each catalog. Users and service principals still need the appropriate Unity Catalog privileges. The arrows between workspaces and storage show workloads reading and writing cloud object storage using Unity Catalog storage credentials, subject to those privileges.

See Unity Catalog best practices.

Define groups for each role​

Define groups in your identity provider, and provision them to your Databricks account, so that they stay consistent with your organization's group definitions. See Groups.

The following groups cover the roles on most Databricks projects. Rename them to match your organization's naming conventions, and keep the boundaries between the roles:

Group

Role

Typical access

Metastore admins

Unity Catalog metastore administration, and creation of catalogs, external locations, and connections

Metastore ownership and privileges to manage objects

Workspace admins

Workspace user, settings, and permission management

Admin access in one workspace

Data engineers

Project pipeline development and maintenance

Ownership of the project's schemas, with read and write access

Data scientists

Machine learning model and AI agent training and deployment

Read access to the project's silver and gold schemas

Data analysts

Business metrics, analyses, and dashboard development

Read access to the project's gold schema

Business users

Viewing dashboards and using Genie Agents

Access to published dashboards and the tables that each Genie Agent uses

Group

Role

Typical access

Metastore admins

Unity Catalog metastore administration, and creation of catalogs, external locations, and connections

Metastore ownership and privileges to manage objects

Workspace admins

Workspace user, settings, and permission management

Admin access in one workspace

Data engineers

Project pipeline development and maintenance

Ownership of the project's schemas, with read and write access

Data scientists

Machine learning model and AI agent training and deployment

Read access to the project's silver and gold schemas

Data analysts

Business metrics, analyses, and dashboard development

Read access to the project's gold schema

Business users

Viewing dashboards and using Genie Agents

Access to published dashboards and the tables that each Genie Agent uses

Create a separate admin group for each workspace, such as dev-ws-admins, staging-ws-admins, and prod-ws-admins, so that admin access to development doesn't extend to production. For the metastore admin group, see Assign a group as the metastore admin. To create the groups themselves, see Create groups for access.

Troubleshooting​

Teams that skip or defer the namespace decision tend to encounter problems that are inexpensive to avoid now and costly to fix after tables and queries depend on them.

Pipelines use the default catalog without a governance plan

If you skip this step, your team builds pipelines directly in the default catalog. Over time, the workspace accumulates unstructured tables in multiple locations.

A small team has an overly complex catalog structure

A five-person team doesn't need catalogs with business unit prefixes. Start with the small organization pattern. If the team grows, you can add prefixes and split catalogs later.

Groups are created inside a workspace

A workspace-local group stays in the workspace where you create it. You can't assign it to other workspaces or grant it access to data in Unity Catalog. Provision account groups from your identity provider using automatic identity management. If your identity provider doesn't support it, use System for Cross-domain Identity Management (SCIM) provisioning instead. Use groups from your identity provider as the source of truth. To migrate legacy workspace-local groups, see Manage workspace-local groups (legacy).

Choose a pattern​

Both of the following patterns use catalogs for each environment, medallion schemas for each project, and schema-level grants to groups. The medium and large pattern adds a business unit prefix to workspaces, catalogs, and groups.

Choose the pattern that matches your organization:

Pattern

Description

Guide

Small organizations

One or two data teams in one business unit, with one catalog for each environment

Organize data for a small organization

Medium and large organizations

Multiple business units in one Databricks account, with one catalog for each business unit and environment, and prefixed groups

Organize data for medium and large organizations

Pattern

Description

Guide

Small organizations

One or two data teams in one business unit, with one catalog for each environment

Organize data for a small organization

Medium and large organizations

Multiple business units in one Databricks account, with one catalog for each business unit and environment, and prefixed groups

Organize data for medium and large organizations

If you start with the small organization pattern, you can add business unit catalogs and groups when a second business unit joins.

These patterns are adapted from Modeling a Medallion Architecture on Unity Catalog for your Organizational Structure.

Next steps​