Skip to main content

Starter Journey: Set up Databricks for production

Use the Starter Journey to set up Databricks for the rest of your organization as an account admin or a member of a platform team. The journey takes a new account to a production-ready environment, and each section covers a decision before later sections depend on it.

By the end of the journey, you have the following:

  • Development, staging, and production workspaces in a cloud tenant layout that you chose, with single sign-on, users and groups synced from your identity provider, and a group as the metastore admin.
  • Catalogs for each environment, bound to their workspaces, with schemas and group grants that follow one naming convention.
  • Data connected through external locations and Lakeflow Connect, and a Lakeflow pipeline that writes bronze and silver tables. Metric views in the Genie Ontology section are the gold layer.
  • Usage dashboards built on system tables, custom tags on compute and serverless usage, and budgets with alerts.
  • Patterns for metric views, dashboards, Genie Agents, models, and agents, with scheduled jobs, row filters and column masks, and continuous integration and continuous delivery (CI/CD) with Declarative Automation Bundles.

The journey puts first the decisions that are hardest to change later: how you split cloud tenants, how you lay out workspaces, and how you organize catalogs. Each page explains what to do and why, and links to the detailed documentation. Follow the journey end to end, or go directly to the section that you need. If you want to try Databricks rather than set it up for a team, see Sign up for Databricks.

Why the foundation matters​

Dashboards, machine learning models, and AI agents all read from the same tables. When those tables are wrong, the errors spread to everything that uses them: reports mislead, models train on bad data, and agents give inaccurate answers. On Databricks, analytics, machine learning, and AI share one platform and one catalog, so a governed data foundation benefits all of them.

Create workspaces, add users and groups, and connect data governed by UC. This shared foundation supports data pipelines, analytics, machine learning, and AI agents.

The first half of the journey builds that foundation: workspaces, identity, a governance structure, data access, and a first pipeline. The second half builds on it. The branches in the diagram are capabilities that share the foundation. Complete them in any order, and choose the ones that match your use cases.

Journey sections​

You must have a Databricks account to start.

Build the foundation​

Complete these sections in order, because later sections assume that the earlier sections are complete:

Section

Outcome

Learn the Databricks foundations and Plan your cloud tenant layout

An understanding of accounts, workspaces, and metastores, and a cloud tenant layout

Set up your Databricks infrastructure

Development, staging, and production workspaces, with single sign-on, users, groups, a metastore admin group, and cost monitoring

Plan your data governance strategy

A catalog, schema, and group structure that fits your organization, existing objects transferred to the right owners, and access control with governed tags, attribute-based access control (ABAC) policies, row filters, and column masks

Connect Databricks to your data

Cloud object storage, databases, and software as a service (SaaS) applications connected to Unity Catalog, a first pipeline that writes bronze and silver tables, SQL editor queries on its silver tables, and a scheduled job in Lakeflow Jobs that runs the pipeline

Section

Outcome

Learn the Databricks foundations and Plan your cloud tenant layout

An understanding of accounts, workspaces, and metastores, and a cloud tenant layout

Set up your Databricks infrastructure

Development, staging, and production workspaces, with single sign-on, users, groups, a metastore admin group, and cost monitoring

Plan your data governance strategy

A catalog, schema, and group structure that fits your organization, existing objects transferred to the right owners, and access control with governed tags, attribute-based access control (ABAC) policies, row filters, and column masks

Connect Databricks to your data

Cloud object storage, databases, and software as a service (SaaS) applications connected to Unity Catalog, a first pipeline that writes bronze and silver tables, SQL editor queries on its silver tables, and a scheduled job in Lakeflow Jobs that runs the pipeline

Deliver analytics and AI​

Choose the sections that match your use cases:

Section

Outcome

Answer business questions with Genie Ontology

Key performance indicators (KPIs) defined as metric views, and served through dashboards, Genie Agents, and domains that Genie One uses as context

Machine learning and MLOps

Feature tables, models registered in Unity Catalog, and batch scoring

Build AI agents on Databricks

AI agents that use your governed data

Section

Outcome

Answer business questions with Genie Ontology

Key performance indicators (KPIs) defined as metric views, and served through dashboards, Genie Agents, and domains that Genie One uses as context

Machine learning and MLOps

Feature tables, models registered in Unity Catalog, and batch scoring

Build AI agents on Databricks

AI agents that use your governed data

Run in production​

Complete this section before your teams depend on the workloads that they build:

Section

Outcome

Operations and CI/CD

Projects that are version controlled and deployed with Declarative Automation Bundles and CI/CD

Section

Outcome

Operations and CI/CD

Projects that are version controlled and deployed with Declarative Automation Bundles and CI/CD

Next steps​