Skip to main content

Organize data for medium and large organizations

Use this pattern when multiple business units, such as finance and marketing, share a Databricks account. It keeps the environments, medallion schemas, and schema-level grants of the small organization pattern, and adds a business unit prefix to every workspace, catalog, and group. The prefix makes each department's data boundary visible in every name. Teams use it to avoid mixing development data, and reviewers can spot grants that cross a boundary.

Add a business unit prefix​

Choose a prefix for each business unit, and add it to the names of workspaces, catalogs, and groups. The examples on this page use an abbreviation for workspaces, such as mkt, and the full name for catalogs and groups, such as marketing:

Resource

Example

Meaning

Workspace

mkt_dev

Marketing development workspace

Catalog

marketing_dev

Marketing development catalog

Group

marketing-de

Data engineers on marketing projects

Resource

Example

Meaning

Workspace

mkt_dev

Marketing development workspace

Catalog

marketing_dev

Marketing development catalog

Group

marketing-de

Data engineers on marketing projects

Define the prefixes one time, document them where every team can find them, and use them consistently. If one team names a catalog fin_dev and another names a catalog finance_dev for the same business unit, the names no longer show the boundary.

Create catalogs for each business unit and environment​

Create a development, staging, and production catalog for each business unit, and bind each catalog to the matching workspace. To create catalogs, you need CREATE CATALOG on the metastore. See Create catalogs and Workspace-catalog binding.

Marketing and finance catalogs for development, staging, and production, each bound to the matching business unit workspace.

The following example uses two business units:

  • Marketing: marketing_dev, marketing_stg, and marketing_prod, bound to the mkt_dev, mkt_stg, and mkt_prod workspaces.
  • Finance: finance_dev, finance_stg, and finance_prod, bound to the fin_dev, fin_stg, and fin_prod workspaces.

Each environment has the same purpose as in the small organization pattern: development for experimentation, staging for validation, and production for trusted data that only service principals write to.

Create medallion schemas for each project​

Within each catalog, create bronze, silver, and gold schemas for each project. For example, a revenue forecasting project in finance_dev uses revenue_bronze, revenue_silver, and revenue_gold.

Medallion schemas for each project in the marketing and finance catalogs.

Grant access to prefixed groups​

Create groups for each business unit, and grant access with the same rules as the small organization pattern: the group that builds a project owns its schemas, owners grant other groups access at the schema level, and service principals write to production.

Permissions on the marketing Customer 360 schemas: marketing-de owns each schema and has Data Editor access, marketing-ml has Data Reader access to silver and gold, and marketing-bi has Data Reader access to gold.

For example, marketing-de owns c360_bronze, c360_silver, and c360_gold in marketing_dev. It grants marketing-ml read access to the silver and gold schemas, and marketing-bi read access to the gold schema.

Create a prefixed group for each role in each business unit. A single unprefixed group, such as data-engineers, with access to every business unit's catalogs removes the boundary that the prefixes create.

Share data across business units​

When one business unit needs another's data, grant access to the source schema instead of copying the tables. Copies drift out of sync with the source, use additional storage, and break lineage. For example, when finance analysts need the marketing Customer 360 data, the owners of marketing_prod.c360_gold grant the finance analyst group access to that schema:

SQL
GRANT USE CATALOG ON CATALOG marketing_prod TO `finance-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA marketing_prod.c360_gold TO `finance-analysts`;

The group and catalog names show that the grant crosses a business unit boundary, which makes it visible during an audit.

If the catalog is bound to specific workspaces, users can access it only from those workspaces. To let finance users query the catalog from a finance workspace, add that workspace to the catalog's bindings with read-only access. See Workspace-catalog binding.

Next steps​