Create compute resources

Preview

Unity Catalog is in Public Preview. To participate in the preview, contact your Databricks representative.

This article shows how to create a Data Science & Engineering or Databricks Machine Learning cluster or a Databricks SQL warehouse that can access data in Unity Catalog.

Requirements

  • Your Databricks account must be on the Premium plan.

  • In a workspace, you must have permission to create compute resources.

Create a Data Science & Engineering cluster

A Data Science & Engineering cluster is designed for running general workloads, such as notebooks.

To create a Data Science & Engineering cluster that can access Unity Catalog:

  1. Log in to the workspace as a workspace-level admin.

  2. Click compute icon Compute.

  3. Click Create cluster.

    1. Enter a name for the cluster.

    2. Set Databricks runtime version to Runtime: 10.3 (Scala 2.12, Spark 3.2.1) or higher.

  4. Click Advanced Options. Set Security Mode to User Isolation or Single User.

    User Isolation clusters can be shared by multiple users, but only SQL workloads are supported. Some advanced cluster features such as library installation, init scripts, and the DBFS Fuse mount are also disabled to ensure security isolation among cluster users.

    To use those advanced cluster features or languages or to run workloads using Python, Scala and R, set the cluster mode to Single User. Single User cluster can also run SQL workloads. The cluster can be used exclusively by a single user (by default the owner of the cluster); other users cannot attach to the cluster. Automated jobs should run in this mode, and the job’s owner should be the cluster’s owner. In this mode, view security cannot be enforced. A user selecting from a view executes with their own permissions.

    For more information about the features available in each security mode, see Cluster security mode.

  5. Click Create Cluster.

Create a Databricks Machine Learning cluster

A Databricks Machine Learning cluster is purpose-built for machine-learning workloads. You can optionally create a GPU-enabled Databricks Machine Learning cluster.

To create a Databricks Machine Learning cluster that can access Unity Catalog:

  1. Log in to the workspace as a workspace-level admin.

  2. In the Data Science & Engineering or Databricks Machine Learning persona, click compute icon Compute.

  3. Click Create cluster.

    1. Enter a name for the cluster.

    2. For Databricks runtime version:

      1. Click ML.

      2. Select either 10.3 ML (Scala 2.12, Spark 3.2.1) or higher, or 10.3 ML (GPU, Scala 2.12, Spark 3.2.1) or higher.

  4. Click Advanced Options. Set Security Mode to User Isolation or Single User. To run Python code, you must use Single User.

    User Isolation clusters can be shared by multiple users, but only SQL workloads are supported. Some advanced cluster features such as library installation, init scripts, and the DBFS Fuse mount are also disabled to ensure security isolation among cluster users.

    To use those advanced cluster features or languages or to run workloads using Python, Scala and R, set the cluster mode to Single User. Single User cluster can also run SQL workloads. The cluster can be used exclusively by a single user (by default the owner of the cluster); other users cannot attach to the cluster. Automated jobs should run in this mode, and the job’s owner should be the cluster’s owner. In this mode, view security cannot be enforced. A user selecting from a view executes with their own permissions.

    For more information about the features available in each security mode, see Cluster security mode.

  5. Click Create Cluster.

Create a Databricks SQL warehouse

A Databricks SQL warehouse is required to run workloads in Databricks SQL, such as queries, dashboards, and visualizations.

To create a SQL warehouse that can access Unity Catalog data:

  1. Log in to the workspace as a workspace-level admin.

  2. From the persona switcher, select SQL.

  3. Click Create, then select SQL Warehouse.

  4. Under Advanced Settings set Channel to Preview.

  1. (Optional) Configure the SQL warehouse as a Serverless SQL warehouse (Preview).

    Serverless SQL warehouses start within seconds, rather than minutes. For more information, see Serverless compute.

SQL warehouses are automatically created with the correct security mode, with no configuration required.