Cluster Tags

Cluster tags allow you to easily monitor the cost of cloud resources used by various groups in your organization. You can specify tags as key-value pairs when you create a cluster, and Databricks applies these tags to cloud resources like VMs and disk volumes.

For convenience, Databricks applies four default tags to each cluster: Vendor, Creator, ClusterName, and ClusterId. You can add custom tags when you create a cluster.

To configure cluster tags:

  1. On the cluster configuration page, click the Advanced Options toggle.

  2. At the bottom of the page, click the Tags tab.

    ../../_images/tags.png
  3. Add a key-value pair for each custom tag.

Enforcing mandatory tags

To ensure that certain tags are always populated when clusters are created, you can apply a specific IAM policy to your account’s primary IAM role (the one created during account setup; contact your AWS administrator if you need access). The IAM policy should include explicit Deny statements for mandatory tag keys and optional values. Cluster creation will fail if required tags with one of the allowed values aren’t provided.

For example, if you want to enforce Department and Project tags, with only specified values allowed for the former and a free-form non-empty value for the latter, you could apply an IAM policy like this one:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MandateLaunchWithTag1",
      "Effect": "Deny",
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:region:accountId:instance/*",
      "Condition": {
        "StringNotEqualsIgnoreCase": {
          "aws:RequestTag/Department": [
              "Deptt1", "Deptt2", "Deptt3"
          ]
        }
      }
    },
    {
      "Sid": "MandateLaunchWithTag2",
      "Effect": "Deny",
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:region:accountId:instance/*",
      "Condition": {
        "StringNotLike": {
          "aws:RequestTag/Project": "?*"
        }
      }
    }
  ]
}

Both ec2:RunInstances and ec2:CreateTags actions are required for each tag for there to be effective coverage of scenarios in which there are clusters that have only on-demand instances, only spot instances, or both.

Tip

We recommend that you add a separate policy statement for each tag. The overall policy might become a bit verbose, but it is easier to debug. See the IAM Policy Condition Operators Reference for a list of operators that can be used in a policy.

Note

Cluster creation errors due to an IAM policy show an encoded error message, starting with Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster. The message is encoded because the details of the authorization status can constitute privileged information that the user who requested the action should not see. See DecodeAuthorizationMessage API (or CLI) for information about how to decode such messages.