Manage serverless compute policies
Serverless compute policies let workspace admins control access to supported serverless compute products, limit their configuration, and enforce tags on newly created compute resources. This page explains how to create policies, grant access, and use them with Lakebase.
Serverless compute policies are in Private Preview and are managed through the REST API. The preview supports Lakebase. Contact your Databricks account team to request access.
Requirements
- Your workspace must be enrolled in the serverless compute policy preview and have access to Lakebase. See Region availability for Lakebase region availability.
- You must be a workspace admin to create a policy. To read or use an existing policy, you need
CAN_USEorCAN_MANAGEon that policy. To update or delete a custom policy, or manage a policy's permissions, you needCAN_MANAGE. - The examples use
curl,jq, and a workspace OAuth token. The authentication example uses the Databricks CLI.
How serverless compute policies work
A serverless compute policy belongs to a workspace. Its allowed_compute_products object specifies which compute products a user can use through the policy and the limits on each product's configuration. Grant access to a policy to let users use the products it allows within those limits.
After creation, the attached policy governs changes to the compute's configuration while the policy exists. Tightening a policy does not change existing compute. Subsequent updates to settings governed by the policy must satisfy its current constraints. For example, a Lakebase capacity change must fit within the policy's current maximum, measured in compute units (CU).
For policies that control classic compute, see Create and manage classic compute policies.
Permissions for configuration changes
Updating a resource's settings or replacing its compute policy requires the following permissions:
Configuration change | Required permissions |
|---|---|
Change settings outside the policy's scope | Permission to manage the resource. Permission to use the policy is not required. |
Change compute settings governed by the policy | Permission to manage the resource and |
Replace the attached policy | Permission to manage the resource and |
The permission to manage a resource depends on the product. For example, a user with CAN_MANAGE on a Lakebase project can change its display name without access to its compute policy. Changing compute capacity or scale-to-zero settings requires permission to use the policy, even when the policy sets no capacity or idle-time limits.
Permission to use a policy does not bypass its constraints. Updated settings must satisfy the policy's limits and preserve its required tag key-value pairs.
You cannot remove a policy from a resource without assigning a replacement. For a Lakebase project, the replacement policy must allow Lakebase. Clearing spec.compute_policy_id assigns the Default Lakebase Policy as the replacement and requires the same permissions.
Policy tags
Use the optional top-level tags map to specify key-value tags, alongside display_name and allowed_compute_products. Tag keys and values are strings. A policy can have up to 20 tags, and each key must be nonempty.
Tags from a serverless compute policy propagate to compute created with that policy. The policy enforces these tags while the policy exists and remains attached. Changes to the compute's tags must preserve the key-value pairs required by the attached policy.
Serverless usage policies, also called serverless budget policies, assign tags for cost attribution and use a separate policy ID. You can use a serverless usage policy together with a serverless compute policy. The created compute receives the union of both policies' tags. For example, if the compute policy specifies environment=development and the usage policy specifies cost_center=engineering, the compute receives both tags.
Lakebase policy settings
To allow Lakebase through a serverless compute policy, include lakebase in allowed_compute_products.
Lakebase project creation
Creating a Lakebase project requires CAN_USE or CAN_MANAGE on a policy that allows Lakebase. When you create a Lakebase project, you automatically receive CAN_MANAGE on it.
Policy permissions do not grant access to existing Lakebase projects or their data. Lakebase project permissions control resource management, with additional policy permissions required to change compute settings governed by a policy. Postgres roles and grants control database access.
Capacity and idle-time limits
Set either or both of the following fields under allowed_compute_products.lakebase:
Field | Description |
|---|---|
| Maximum capacity in Lakebase CU. For an autoscaling compute, the configured maximum must fit within this limit. |
| Maximum inactivity timeout, in minutes, before compute scales to zero. A compute subject to this limit must have scale-to-zero enabled, with a timeout at or below the limit. |
Both fields accept positive integers. Omitting a field does not set a policy limit for that setting. An empty lakebase object allows Lakebase without either limit. A policy must include a lakebase object to allow Lakebase.
For example, the following lakebase object limits each compute to 16 CU and requires scale to zero after at most 15 minutes of inactivity:
{
"max_capacity_units_per_instance": 16,
"max_idle_minutes_before_scale_to_zero": 15
}
These limits supplement the supported compute sizes and scale-to-zero settings. A policy does not make an unsupported configuration available. See Limitations for configurations that cannot use scale to zero.
Default Lakebase Policy
The workspace's Default Lakebase Policy allows Lakebase without capacity or idle-time limits. Workspace users can use this policy to create Lakebase projects by default.
Workspace admins can grant or revoke permissions on the Default Lakebase Policy through the Permissions API. They cannot delete the policy or change its name or configuration.
When you create a Lakebase project without specifying a compute policy, the request uses the Default Lakebase Policy. You must have permission to use that policy. Creating a custom policy does not revoke access to the default policy or automatically apply the custom policy to existing Lakebase projects. To require users to select a custom policy, see Require a custom policy for Lakebase project creation.
Set up API authentication
Use your workspace URL and a workspace OAuth token for these requests. See Authorize user access to Databricks with OAuth for authentication details.
-
Set
WORKSPACEto your workspace URL, without a trailing slash:Bashexport WORKSPACE="https://<workspace-host>" -
Authenticate to that workspace:
Bashdatabricks auth login --host "$WORKSPACE" -
Retrieve an OAuth token for the authenticated identity:
Bashexport DATABRICKS_TOKEN="$(databricks auth token --host "$WORKSPACE" | jq -r .access_token)"
Refresh the token if it expires. Run each request as an identity with the required permissions.
Create a policy
The following example creates a serverless compute policy that allows Lakebase:
-
As a workspace admin, send the following request. It creates a policy with a 16 CU maximum, a 15-minute maximum inactivity timeout, and an enforced
environment=developmenttag:Bashcurl --fail-with-body -sS -X POST "$WORKSPACE/api/2.0/policies/compute" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d '{"display_name": "Lakebase development","allowed_compute_products": {"lakebase": {"max_capacity_units_per_instance": 16,"max_idle_minutes_before_scale_to_zero": 15}},"tags": {"environment": "development"}}' -
Copy the
compute_policy_idfrom the response and set it for subsequent requests:Bashexport COMPUTE_POLICY_ID="<compute-policy-id>" -
Retrieve the policy to verify its name and product constraints. This response does not include the policy's tags:
Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/policies/compute/$COMPUTE_POLICY_ID" \-H "Authorization: Bearer $DATABRICKS_TOKEN"
The request body is the policy object itself. The service assigns compute_policy_id. Choose a unique display name in the workspace, from 1 to 127 characters.
Grant access to a policy
Use the Permissions API with the object type compute-policy. The policy creator has CAN_MANAGE.
-
As an identity with
CAN_MANAGE, grant an existing workspace group permission to use the policy. Replacelakebase-developerswith your group name:Bashcurl --fail-with-body -sS -X PATCH \"$WORKSPACE/api/2.0/permissions/compute-policy/$COMPUTE_POLICY_ID" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d '{"access_control_list": [{"group_name": "lakebase-developers","permission_level": "CAN_USE"}]}' -
Verify the grant:
Bashcurl --fail-with-body -sS \"$WORKSPACE/api/2.0/permissions/compute-policy/$COMPUTE_POLICY_ID" \-H "Authorization: Bearer $DATABRICKS_TOKEN"
Use user_name to grant access to a user, or service_principal_name with the application ID of a Databricks service principal. Grant CAN_MANAGE to let a principal use the policy and manage its permissions. For custom policies, CAN_MANAGE also allows updating the configuration and deleting the policy. PATCH updates the specified grants. PUT replaces the direct permission assignments, so include the full set you intend to retain.
Require a custom policy for Lakebase project creation
As a workspace admin, use the following steps to require users to select a custom policy:
- List policies, following any
next_page_tokenvalues. Find the Default Lakebase Policy by itsis_defaultvalue andlakebaseentry inallowed_compute_products. Copy itscompute_policy_id. - Grant the affected users or groups access to a custom policy that allows Lakebase.
- For existing Lakebase projects that use the default policy, replace their policy if users need to continue changing governed compute settings. Revoking default-policy access prevents those changes on compute that still uses it.
- Retrieve the default policy's permissions with
GET /api/2.0/permissions/compute-policy/<default-policy-id>. Review both user and group grants. The default policy initially grants access to theusersgroup, so removing a direct user grant alone does not remove that user's access. - Replace the direct permission assignments with
PUT /api/2.0/permissions/compute-policy/<default-policy-id>. Include the direct grants you intend to retain inaccess_control_list, and omit grants that give the affected users access. Replace the broadusersgrant with grants to specific groups if some users should retain default-policy access. Each request entry uses a principal name andpermission_level, as in the grant example. Do not send the permissions GET response unchanged as the PUT body. - As an affected user, list the policies again. Verify that the custom policy is available and the default policy is no longer available. The user must now specify a custom policy ID when creating a Lakebase project.
Use a policy with Lakebase projects
Use the Postgres API (/api/2.0/postgres/) to create and manage Lakebase projects. A Lakebase project contains branches, and compute endpoints provide compute for those branches. See Computes and endpoints.
Select a compute policy with spec.compute_policy_id when you create a Lakebase project. The attached policy governs compute configuration and enforces its required tags on the associated compute. Policy management uses the separate /api/2.0/policies/compute API.
Create a Lakebase project with a policy
Use an identity with CAN_USE or CAN_MANAGE on the policy created above:
-
Create a Lakebase project with a 2–8 CU autoscaling range and a 15-minute inactivity timeout:
Bashcurl --fail-with-body -sS -X POST \"$WORKSPACE/api/2.0/postgres/projects?project_id=policy-example" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d "$(jq -n --arg policy_id "$COMPUTE_POLICY_ID" '{"spec": {"display_name": "Policy example","compute_policy_id": $policy_id,"default_endpoint_settings": {"autoscaling_limit_min_cu": 2,"autoscaling_limit_max_cu": 8,"suspend_timeout_duration": "900s"}}}')"The capacity values are Lakebase CU. The maximum of 8 CU fits the policy's 16 CU limit. Lakebase expresses the inactivity timeout as a duration in seconds:
"900s"equals the policy's 15-minute limit. These defaults configure the initial compute and provide the starting settings for new computes in the Lakebase project.To also apply a serverless usage policy, add
"budget_policy_id": "<usage-policy-id>"insidespec, using the ID of a policy you can use. Compute receives tags from both policies. -
Copy the operation
namefrom the response, and set it for the status request:Bashexport OPERATION_NAME="<operation-name>" -
Retrieve the operation until
doneistrue:Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/postgres/$OPERATION_NAME" \-H "Authorization: Bearer $DATABRICKS_TOKEN"If the response contains
error, resolve it before continuing. Create, update, and delete requests use this long-running operation pattern.
Read and list Lakebase projects
-
Retrieve the Lakebase project:
Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/postgres/projects/policy-example" \-H "Authorization: Bearer $DATABRICKS_TOKEN" -
Check that
status.compute_policy_idmatches the policy you selected. -
List the Lakebase projects you can access:
Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/postgres/projects" \-H "Authorization: Bearer $DATABRICKS_TOKEN"
If the list response includes next_page_token, pass that value as page_token to retrieve the next page.
Update a Lakebase project
Use an identity with CAN_MANAGE on the Lakebase project. Changing its display name does not require permission to use its compute policy.
-
Update the Lakebase project's display name, specifying the field in the update mask:
Bashcurl --fail-with-body -sS -X PATCH \"$WORKSPACE/api/2.0/postgres/projects/policy-example?update_mask=spec.display_name" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d '{"spec": {"display_name": "Lakebase development"}}' -
Wait for the returned operation to complete, then retrieve the Lakebase project to verify the change.
Changing a Lakebase project's capacity or scale-to-zero compute defaults also requires CAN_USE or CAN_MANAGE on the policy, even if it sets no limit for the setting. The new values must satisfy any applicable policy limits. Defaults apply when new computes are created. To change an existing compute, update its endpoint as shown in the following procedure.
To change the attached policy, see Replace a Lakebase project's policy.
Update Lakebase compute
The following example raises the existing compute's maximum to 16 CU and shortens its inactivity timeout to 10 minutes. You need CAN_MANAGE on the Lakebase project and CAN_USE or CAN_MANAGE on the policy:
-
List the endpoints on the Lakebase project's
productionbranch:Bashcurl --fail-with-body -sS \"$WORKSPACE/api/2.0/postgres/projects/policy-example/branches/production/endpoints" \-H "Authorization: Bearer $DATABRICKS_TOKEN" -
Copy the initial compute's endpoint
namefrom the response. The name includes the Lakebase project, branch, and endpoint path:Bashexport ENDPOINT_NAME="<endpoint-name>" -
Update the maximum CU and scale-to-zero settings. Use
spec.suspensionin the update mask for the inactivity timeout:Bashcurl --fail-with-body -sS -X PATCH \"$WORKSPACE/api/2.0/postgres/$ENDPOINT_NAME?update_mask=spec.autoscaling_limit_max_cu,spec.suspension" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d '{"spec": {"autoscaling_limit_max_cu": 16,"suspend_timeout_duration": "600s"}}' -
Wait for the returned operation to complete, then retrieve the endpoint:
Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/postgres/$ENDPOINT_NAME" \-H "Authorization: Bearer $DATABRICKS_TOKEN"Verify
status.autoscaling_limit_max_cuandstatus.suspend_timeout_duration.
The updated maximum fits the policy's 16 CU limit, and "600s" is below its 15-minute idle limit. For this compute, requests above 16 CU, inactivity timeouts above 15 minutes, or disabling the scale-to-zero setting fail policy validation. Choose settings within the policy's limits or ask a policy manager to review them.
Replace a Lakebase project's policy
You need CAN_MANAGE on the Lakebase project and CAN_USE or CAN_MANAGE on both the current policy and its replacement. The replacement must allow Lakebase.
Switching policies is a configuration update. The Lakebase project's existing compute and compute defaults must satisfy the replacement policy's constraints before the update can complete.
-
Select a replacement policy and set its ID:
Bashexport REPLACEMENT_COMPUTE_POLICY_ID="<replacement-policy-id>" -
Verify that the Lakebase project's compute defaults and existing compute satisfy the replacement policy's capacity and idle-time limits. Update existing compute and adjust the defaults if necessary.
-
Update the Lakebase project's policy assignment:
Bashcurl --fail-with-body -sS -X PATCH \"$WORKSPACE/api/2.0/postgres/projects/policy-example?update_mask=spec.compute_policy_id" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d "$(jq -n --arg policy_id "$REPLACEMENT_COMPUTE_POLICY_ID" '{"spec": {"compute_policy_id": $policy_id}}')" -
Wait for the returned operation to finish with
done: true. If it contains anerror, resolve it before continuing. -
Retrieve the Lakebase project and verify that
status.compute_policy_idmatches the replacement policy ID:Bashcurl --fail-with-body -sS "$WORKSPACE/api/2.0/postgres/projects/policy-example" \-H "Authorization: Bearer $DATABRICKS_TOKEN"
Clearing spec.compute_policy_id selects the Default Lakebase Policy and requires permission to use it. It does not leave the Lakebase project without a policy.
Delete a Lakebase project
By default, deleting a Lakebase project makes it unavailable and retains it for recovery during the retention period. Deleting a Lakebase project does not delete the workspace policy.
-
Delete the example Lakebase project:
Bashcurl --fail-with-body -sS -X DELETE \"$WORKSPACE/api/2.0/postgres/projects/policy-example" \-H "Authorization: Bearer $DATABRICKS_TOKEN" -
Wait for the returned operation to complete. For deletion requirements, including protected branches, see Delete a project.
Manage policies
Use the following operations to inspect and maintain policies in your workspace.
Get and list policies
Get an individual policy with GET /api/2.0/policies/compute/{compute_policy_id}, as shown in Create a policy. To list policies you can use or manage, run:
curl --fail-with-body -sS "$WORKSPACE/api/2.0/policies/compute?page_size=100" \
-H "Authorization: Bearer $DATABRICKS_TOKEN"
The response contains compute_policies and, when more results are available, next_page_token. Pass that token as page_token in the next request. The default page size is 100, and the maximum is 150. Results are not sorted. Each policy includes can_use and can_manage for the calling identity. is_default identifies a default policy.
Update a policy
Policy updates require an update_mask query parameter. Supported paths are display_name and allowed_compute_products. To update both, separate the paths with a comma. This endpoint does not support updating tags.
-
Retrieve the policy and review the settings you intend to retain.
-
Send the changed fields. For example, raise the example policy's capacity limit to 32 CU and retain its 15-minute idle limit:
Bashcurl --fail-with-body -sS -X PATCH \"$WORKSPACE/api/2.0/policies/compute/$COMPUTE_POLICY_ID?update_mask=allowed_compute_products" \-H "Authorization: Bearer $DATABRICKS_TOKEN" \-H "Content-Type: application/json" \-d '{"allowed_compute_products": {"lakebase": {"max_capacity_units_per_instance": 32,"max_idle_minutes_before_scale_to_zero": 15}}}' -
Check the returned policy to verify the updated limit.
Updating the lakebase object replaces its constraints. Include every Lakebase constraint you intend to retain. For example, omitting an existing idle-time limit removes that limit. An empty lakebase object clears its limits while continuing to allow Lakebase. Nested mask paths, such as allowed_compute_products.lakebase.max_capacity_units_per_instance, are not supported.
To rename a policy without changing its constraints, use update_mask=display_name and a body such as {"display_name": "Lakebase application policy"}.
Lowering a policy's capacity limit from 16 to 8 CU leaves existing 16-CU compute unchanged. Any later capacity change must comply with the 8-CU limit. When switching to a different policy, existing compute must satisfy that policy's constraints before the update can complete.
Delete a policy
Deleting a policy does not delete compute created using that policy. Enforcement of its constraints depends on the policy continuing to exist and remaining attached.
Before deleting a custom policy:
- List Lakebase projects, following any
next_page_tokenvalues. Find Lakebase projects whosestatus.compute_policy_idmatches the policy you plan to delete. - Confirm that the inventory covers all affected Lakebase projects.
CAN_MANAGEon a policy does not grant access to every Lakebase project that uses it. Coordinate with Lakebase project owners or a workspace admin for projects you cannot view. - Replace the policy on affected Lakebase projects before deleting it if those projects must remain governed by a policy.
To delete a custom policy, run:
curl --fail-with-body -sS -X DELETE \
"$WORKSPACE/api/2.0/policies/compute/$COMPUTE_POLICY_ID" \
-H "Authorization: Bearer $DATABRICKS_TOKEN"
The Default Lakebase Policy cannot be deleted.
Troubleshooting
Error or condition | Action |
|---|---|
| Verify that you have |
| Verify your permission to manage the resource. Changes to compute settings governed by the policy also require |
| Verify your permission to manage the resource and |
| Reduce |
| Enable scale to zero and choose an inactivity timeout at or below the policy limit, on a compute that supports scale to zero. |
| Check that the name is nonblank and at most 127 characters, at least one compute product is present, limits are positive integers, and the update mask contains supported paths. Default policy names and configuration cannot be changed. |
| Choose a different name. Default policy names are reserved. |
Limitations
Policy management
- Policies are scoped to a workspace. Create and grant access to policies in each workspace where you use them.
- The Default Lakebase Policy cannot be deleted or have its name or configuration changed. Workspace admins can grant or revoke permissions on it.
- Removing a policy from a resource requires a replacement. You need permission to manage the resource and
CAN_USEorCAN_MANAGEon both policies. Existing compute must satisfy the replacement policy's constraints before the update can complete. - Tightening a policy does not change existing compute. Later changes to governed settings must satisfy the updated limits.
- Deleting a custom policy does not delete compute created using that policy. Enforcement of that policy's constraints depends on the policy's continued existence.
- The policy API does not return tags in get or list responses, or support changing tags through the policy update endpoint.
Lakebase compute
- Changing compute capacity or scale-to-zero settings requires
CAN_MANAGEon the Lakebase project andCAN_USEorCAN_MANAGEon its policy. This applies even if the policy sets no limit for the setting. - Scale to zero requires a maximum compute size of 32 CU or smaller. If a policy requires an idle-time limit, choose a compatible compute size. See Configure scale to zero.
- High availability uses a compute group with multiple nodes and requires scale to zero to be disabled. The policy's idle-time limit does not apply to these groups. Capacity limits still apply.
- Changing a Lakebase project's compute defaults affects new compute. Update existing compute endpoints separately to change their capacity or idle timeout.