Skip to main content
Last updated on

Create a least-privilege workspace

A least-privilege workspace is a Databricks workspace on Google Cloud that you create by granting Databricks only a narrow, explicitly defined set of IAM permissions. Instead of granting broad roles and letting Databricks automatically provision and configure cloud resources, you create custom IAM roles, set up network and encryption resources yourself, and provision the workspace through the Account API.

This approach is intended for security-conscious and regulated accounts that must minimize the permissions granted to third-party services. Because you perform setup steps manually, creating a least-privilege workspace requires more effort than the standard creation flow.

For most deployments, Databricks recommends the standard workspace creation flow, which uses the account console and broader permissions to provision resources automatically. See Create a classic workspace.

note

This page describes an advanced, API-driven workflow that does not support retries. Complete and verify each step in order. If a step is missed or misconfigured, you must delete the workspace and start over.

How least-privilege workspace creation works​

Least-privilege workspace creation uses two distinct sets of IAM roles, each assigned to a different Google service account:

  • Workspace creator: A Google service account that you own and use to call the Account API. It needs only read-level permissions to validate settings during creation. You grant it the workspace creator roles described in Workspace creator roles.
  • Workspace service account: A Google service account that Databricks creates in its regional control plane and returns to you during workspace creation. You grant it the workspace operator roles so that it can operate and manage the workspace.

Because Databricks returns the workspace service account partway through creation, the flow is split into two phases: you create the workspace in a PROVISIONING state, grant the operator roles to the returned service account, and then update the workspace to bring it to a RUNNING state. You make these calls directly against the Account API and create the supporting Google Cloud resources with the gcloud CLI.

Before you begin​

Complete the following prerequisites before you create a workspace.

Create the required IAM roles​

Before you begin, you must create a series of IAM roles in your GCP projects:

Set up the workspace creator service account​

  1. Create a new Google service account or select an existing one.

  2. Assign the workspace creator IAM role(s) you created to the service account, binding each role on the appropriate project:

    To grant a role on a project, use gcloud projects add-iam-policy-binding:

    Bash
    gcloud projects add-iam-policy-binding <project-id> \
      --member="serviceAccount:<creator-sa-email>" \
      --role="projects/<project-id>/roles/<creator-role-id>"

  3. Add the service account as a user in the Databricks account console.

  4. Assign the service account the Account Admin role in your Databricks account. See Service principals.

Verify the VPC firewall rule​

Confirm that your VPC has the default intra-subnet ingress firewall rule that allows internal communication between VM instances in the same subnet. Describe the rule to confirm it exists:

Bash
gcloud compute firewall-rules describe default-allow-internal \
  --project=<vpc-host-project-id>

The rule must have the following properties, where the source range matches your Databricks subnet range:

  • Direction: INGRESS
  • Priority: 65534
  • Source ranges: your Databricks subnet range
  • Allowed protocols: all

If an equivalent rule does not exist, create one. For more information on default firewall rules, see the GCP documentation.

Configure authentication​

The Account API calls in this page authenticate with Google ID tokens. To keep the commands in this section readable, they use a single-account model: the gcloud CLI authenticates directly as the workspace creator service account, which acts as both the token-creating and resource-owning service account.

note

For production and regulated deployments, Databricks recommends a two-service-account model, in which a token-creating service account (SA-1) impersonates a resource-owning service account (SA-2) rather than holding a long-lived key itself. To apply that model here, treat the workspace creator service account as SA-2 and create a separate SA-1 to impersonate it. For setup steps and token expiration limits, see Authenticate with Google ID tokens.

The API calls in this page use two short-lived tokens. Generate them immediately before you make API calls to avoid expiration:

  • An identity token that Databricks uses to verify the caller (passed in the Authorization header):

    Bash
    gcloud auth print-identity-token --audiences="https://accounts.gcp.databricks.com"

  • A Google OAuth access token for credentials passthrough (passed in the X-Databricks-GCP-SA-Access-Token header):

    Bash
    gcloud auth print-access-token

The examples in this page assume that you have stored these values in the ID_TOKEN and ACCESS_TOKEN environment variables. For details about these headers, see Authenticate with Google ID tokens.

Step 1: Create the network configuration and optional features​

Pre-create your network configuration, along with any optional Private Service Connect or customer-managed keys resources, before you create the workspace. If you use Private Service Connect, register its endpoints first so that you can reference their IDs in the network configuration. Each request returns an ID that you must record and pass to the workspace in Step 5. Each ID also appears in the account console URL for the object.

For example, a network configuration URL has the form:

Text
https://<account-console>/cloud-resources/networking/network-configurations/<network-id>/

(Optional) Register Private Service Connect endpoints​

If you are creating a Private Service Connect-enabled workspace, register the endpoints before you create the network configuration so that you can reference their IDs in it. Create the required Private Service Connect endpoints in Google Cloud and register them with Databricks. For requirements and instructions, see Enable Private Service Connect for your workspace.

If you are not using Private Service Connect, skip to Create the network configuration.

Register the workspace VPC endpoint:

Bash
curl -X POST \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/vpc-endpoints \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "vpc_endpoint_name": "<frontend-endpoint-name>",
    "gcp_vpc_endpoint_info": {
      "project_id": "<vpc-host-project-id>",
      "psc_endpoint_name": "<workspace-psc-endpoint-name>",
      "endpoint_region": "us-central1"
    }
  }'

Register the backend (relay) VPC endpoint:

Bash
curl -X POST \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/vpc-endpoints \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "vpc_endpoint_name": "<backend-endpoint-name>",
    "gcp_vpc_endpoint_info": {
      "project_id": "<vpc-host-project-id>",
      "psc_endpoint_name": "<relay-psc-endpoint-name>",
      "endpoint_region": "us-central1"
    }
  }'

Record the vpc_endpoint_id from each response. When you create the network configuration in the next section, use the frontend ID in its rest_api field and the backend ID in its dataplane_relay field.

Also create a private access settings object:

Bash
curl -X POST \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/private-access-settings \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "private_access_settings_name": "<pas-name>",
    "region": "us-central1",
    "public_access_enabled": true,
    "private_access_level": "ACCOUNT"
  }'

Record the private_access_settings_id from the response.

Create the network configuration​

Create a network configuration that represents your customer-managed VPC and its subnets. For requirements and console instructions, see Configure a customer-managed VPC.

The following request creates a network configuration. The network_name must be 3-30 characters and contain only the characters a-z, A-Z, -, and _. If you use Private Service Connect, include the vpc_endpoints field with the VPC endpoint IDs you recorded in Register Private Service Connect endpoints; otherwise, omit it.

Bash
curl -X POST \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/networks \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "network_name": "<network-name>",
    "gcp_network_info": {
      "network_project_id": "<vpc-host-project-id>",
      "vpc_id": "<vpc-id>",
      "subnet_id": "<subnet-id>",
      "subnet_region": "us-central1"
    },
    "vpc_endpoints": {
      "dataplane_relay": ["<relay-vpc-endpoint-id>"],
      "rest_api": ["<workspace-vpc-endpoint-id>"]
    }
  }'

Record the network_id from the response.

(Optional) Set up customer-managed keys​

As part of pre-creation, if you are creating a customer-managed keys-enabled workspace, set up and register your keys now. You pass the returned key IDs to the workspace later, in Step 5:

  1. Create a Cloud KMS key in Google Cloud.

  2. For workspace storage encryption, grant the Cloud KMS CryptoKey Encrypter/Decrypter role to the default Compute Engine service agent and the default Cloud Storage service agent in the workspace project:

    Bash
    gcloud kms keys add-iam-policy-binding KEY_NAME \
      --keyring=KEY_RING \
      --location=LOCATION \
      --role=roles/cloudkms.cryptoKeyEncrypterDecrypter \
      --member=serviceAccount:service-PROJECT_NUMBER@compute-system.iam.gserviceaccount.com \
      --member=serviceAccount:service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com

  3. Register the key with Databricks for both storage and managed services. For requirements, see Configure customer-managed keys for encryption.

    Bash
    curl -X POST \
      https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/customer-managed-keys \
      -H "Authorization: Bearer $ID_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "gcp_key_info": {
          "kms_key_id": "<kms-key-resource-id>",
          "manual": true
        },
        "use_cases": ["STORAGE", "MANAGED"]
      }'

    Record the customer_managed_key_id from the response.

    Set "manual": true so that Databricks does not attempt to grant itself access to the KMS key. In a least-privilege workspace, you grant this access yourself: the encrypter/decrypter role for storage is granted above, and the role for managed services is granted to the workspace service account later, in Step 4.

Step 2: Create the databricks-compute service account​

Create the databricks-compute service account used by all compute resources in the workspace that do not have a custom service account attached. This service account has minimal permissions, limited to logging and metrics.

Bash
gcloud iam service-accounts create databricks-compute \
  --display-name="Databricks Compute Service Account" \
  --project=<workspace-project-id>

Step 3: Create the workspace​

Create the workspace in a PROVISIONING state by calling the Account API. Authenticate with both the access token and the identity token from Configure authentication. Include only the fields that apply to your workspace. For example, omit the Private Service Connect and customer-managed keys fields if you are not using them.

Bash
curl -X POST \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/workspaces \
  -H "X-Databricks-GCP-SA-Access-Token: $ACCESS_TOKEN" \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "workspace_name": "<workspace-name>",
    "cloud": "gcp",
    "cloud_resource_container": {
      "gcp": {
        "project_id": "<workspace-project-id>"
      }
    },
    "location": "us-central1",
    "pricing_tier": "ENTERPRISE",
    "expected_workspace_status": "PROVISIONING"
  }'

A successful request returns a 200 response that includes the workspace ID and the workspace service account:

JSON
{
  "account_id": "<account-id>",
  "cloud": "gcp",
  "cloud_resource_container": {
    "gcp": {
      "project_id": "my-gcp-project"
    }
  },
  "creation_time": 1643668346544,
  "deployment_name": "1614665312930232.2",
  "location": "us-central1",
  "pricing_tier": "ENTERPRISE",
  "workspace_id": 1614665312930232,
  "workspace_name": "example-workspace-name",
  "workspace_status": "PROVISIONING",
  "workspace_status_message": "Workspace resources are being set up.",
  "workspace_service_account": "<workspace-service-account>"
}

Record the workspace_service_account value from the response. You grant the operator roles to this service account in the next step.

Step 4: Grant roles to the workspace service account​

Grant the workspace operator roles to the workspace service account returned in Step 3, using the workspace_service_account value from that response.

Grant the following roles at the project level on the service (workspace) project:

  • lpw.databricks.project.role.v2
  • lpw.databricks.resource.role.v2. Scope this grant to the workspace by adding an IAM condition on the workspace ID.

For the permissions that each role includes, see Least-privilege workspace operator roles.

Bind the network role to the subnet​

Bind the network role to the workspace service account on the Databricks subnet. This applies to the primary subnet used by the workspace:

Bash
gcloud compute networks subnets add-iam-policy-binding <subnet> \
  --project=<vpc-host-project-id> \
  --region=us-central1 \
  --member="serviceAccount:<workspace-service-account>" \
  --role="projects/<vpc-host-project-id>/roles/lpw.databricks.network.role.v2"

(Optional) Grant the customer-managed keys encrypter/decrypter role for managed services​

If you are using customer-managed keys for managed services, grant the Cloud KMS CryptoKey Encrypter/Decrypter role to the workspace service account to enable encryption for managed services:

Bash
gcloud kms keys add-iam-policy-binding KEY_NAME \
  --keyring=KEY_RING \
  --location=LOCATION \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter \
  --member=serviceAccount:db-WORKSPACEID@db-regional-cp-project.iam.gserviceaccount.com
warning

Stop and verify that you have completed every preceding step before you continue. The workspace creation flow does not support retries. If a step is missed or misconfigured, workspace creation fails and you must delete the workspace and start over.

Step 5: Update the workspace to use your network configuration​

Update the workspace with the network configuration and optional Private Service Connect and customer-managed keys IDs that you recorded in Step 1. This request starts the final provisioning, which then continues asynchronously.

Include only the fields that apply to your workspace. For example, omit the customer-managed keys fields if you are not using them.

Bash
curl -X PATCH \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/workspaces/<workspace-id> \
  -H "X-Databricks-GCP-SA-Access-Token: $ACCESS_TOKEN" \
  -H "Authorization: Bearer $ID_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "network_id": "<network-id>",
    "private_access_settings_id": "<private-access-settings-id>",
    "storage_customer_managed_key_id": "<storage-customer-managed-key-id>",
    "managed_services_customer_managed_key_id": "<managed-services-customer-managed-key-id>"
  }'

A successful request returns a 200 response. The workspace_status is still PROVISIONING, not RUNNING, because the remaining provisioning operations run asynchronously after the request returns:

JSON
{
  "workspace_id": 1614665312930232,
  "workspace_name": "example-workspace-name",
  "workspace_status": "PROVISIONING",
  "workspace_status_message": "Workspace resources are being set up.",
  "account_id": "<account-id>",
  "network_id": "<network-id>",
  "pricing_tier": "ENTERPRISE",
  "location": "us-central1",
  "cloud": "gcp"
}

If the request fails, the response indicates the cause. For example:

  • A 400 BAD_REQUEST error indicates insufficient permissions on the Google Cloud project. Review the operator role grants in Step 4.
  • A 404 RESOURCE_DOES_NOT_EXIST error indicates that a referenced configuration ID, such as the network ID, does not exist. Confirm the IDs that you recorded in Step 1.

Provisioning typically completes within a few minutes. Poll the workspace until its workspace_status becomes RUNNING before you continue:

Bash
curl -X GET \
  https://accounts.gcp.databricks.com/api/2.0/accounts/<account-id>/workspaces/<workspace-id> \
  -H "Authorization: Bearer $ID_TOKEN"

After you create the workspace​

After the workspace reaches the RUNNING state:

  • Configure Domain Name System (DNS) for PSC. If your workspace uses Private Service Connect, complete the DNS configuration. See Enable Private Service Connect for your workspace.
  • Verify the workspace. Confirm that the workspace is configured correctly by starting a Databricks cluster.

Create a catalog backed by your cloud storage​

If a Unity Catalog metastore does not already exist in the same region as the workspace, Databricks creates a metastore and assigns it to the workspace. No default catalog is created automatically.

To make the workspace ready for data, create a catalog backed by your cloud storage:

  1. Create a storage credential. See Create a storage credential that accesses GCS.
  2. Create an external location. See Create an external location for a GCS bucket.
  3. Create a catalog. See Create catalogs.

Next steps​

After you create a least-privilege workspace, you can start building out your data strategy. Databricks recommends the following pages:

On this page