Attribute serverless usage with budget policies

Preview

This article explains how to use budget policies to enforce cost attribution tags on serverless compute workloads.

Budget policies consist of tags that are applied to any serverless compute activity incurred by a user assigned to the policy. The tags are logged in your billing records, allowing you to attribute serverless usage to specific budgets. For more on creating budgets, see Use budgets to monitor account spending.

Budget policy permissions

You must be a workspace admin to create budget policies. Non-admins can manage policies if they are assigned Budget policy: Manager permissions.

Workspace admins can manage and view budget policies they created or the ones they have explicit permissions on. To view and manage all policies for a given account, the workspace admin must additionally have the Billing admin account-level role. This role is assigned in the account console.

By default, new policies can only be accessed by the policy creator and users with the billing admin role.

Because budget policies are account-level resource objects, managers can manage policies from any workspace they have access to.

Create a budget policy

Budget policies are managed in your workspace admin settings. To create a new budget policy:

Click your username in the top bar of the Databricks workspace and select Settings.
Click Compute.
Next to Budget policies, click Manage.
Click Create.
Add a name and your tags then click Create.

Manage budget policy permissions

There are two types of permissions you can grant on your budget policies:

User: A budget policy user can select the policy whenever they create a notebook, job, or pipeline.
Manager: A budget policy manager can edit the policy’s definitions and permissions. Any user in your workspace can be given the manager permissions.

To assign permissions on a budget policy:

On the policy’s page, click the Permissions tab.
Click Grant access.
Select the user, group, or service principal you want to assign the policy to.
Under Role, select the role(s) you want to give the user.
To continue adding identities to the policy, click Add another.
Click Save.

If a user is assigned a single policy, then that policy is applied to their resources by default. If a user is assigned multiple policies, they must select the appropriate policy whenever they create a new serverless notebook, job, or pipeline. If you don’t select a policy, the setting defaults to whichever policy comes first alphabetically.

Update a budget policy

To update an existing budget policy’s name or tags:

Click your username in the top bar of the Databricks workspace and select Settings.
Click Compute.
Next to Budget policies, click Manage.
Select the budget policy you want to update, then click Edit.
Make the updates then click Save.

Policy changes are only applied to usage initiated after the policy update. They are not applied to currently running serverless usage.

Analyze budget policy tags in the billing records

Once a policy is applied to a notebook, job, or Delta Live Tables pipeline, any tags contained in the policy propagate to your system.billing.usage system table in the custom_tags column.

If a notebook is run as part of a job, only the job’s budget policy is applied to the usage record.

Where to select the budget policy

Users assigned to multiple budget policies must select the appropriate policy whenever they create a new notebook, job, pipeline or model serving endpoint.

Notebook: Users can select the budget policy in the Connect > More… drop-down or the Environment panel. See Select a budget policy for your serverless usage.
Job: Users select the budget policy in the Job details section. See Select a budget policy for your serverless usage.
Pipeline: Users select the budget policy when they create or edit the pipeline. See Configure a serverless Delta Live Tables pipeline.
Serving endpoints: Users select the budget policy when they create or edit the model serving endpoint. See Add a budget policy for a model serving endpoint.

Known limitations

Budget policies have the following limitations:

Existing notebooks, jobs, and Delta Live Tables pipelines are not automatically assigned policies after their owners are granted access to a policy. Policies are required on new assets but existing assets must be updated to add a policy.
When creating a scheduled notebook job from the notebook page, the notebook’s budget policy defaults back to the first available policy by alphabetical order.
Policy IDs stored with an asset remain even if the policy is deleted. These policies do not apply any tags.
Updates to budget policy tags are reflected on any serverless usage initiated after the change to the policy. For example, if a job is running while a budget policy is updated, the update does not take effect for the existing job run. The next job run uses the updated tags.
Git repo integration does not have repo-configurable budget policies, the budget policy defaults to the user’s last chosen policy.
Budget policies do not apply tags to classic compute resources.
Tags inherited from budget policies do not appear on the Jobs list UI.
Updates to tags won’t be reflected in new pipeline updates if the pipeline is in Development mode. The changes take 24 hours to propagate.
Pipelines triggered by jobs do not inherit the job’s budget policy. Users must set the pipeline’s budget policy.