Manage notification destinations
This page teaches you how to create and configure notification destinations for your workspace.
System notifications are messages that tell you when your workflow experiences a run event (start, success, and failure). By default, notifications are sent to user email addresses, but admins can configure alternate notification destinations using webhooks or the Databricks Genie app for Slack and Microsoft Teams (Public Preview). This allows you to build event-driven integrations with Databricks.
Admins can also configure notification destinations to receive access requests from workspace users, such as privilege requests on Unity Catalog objects.
You must be a Databricks workspace admin to manage notification destinations. After a destination is configured, it is available to all users.
Account-level events, such as budget alerts, use account-level notification destinations, which account admins configure in the account console. See Manage account-level notification destinations.
Networking requirements
For webhook-based destinations, the receiving endpoint must meet the following requirements:
- Databricks enforces the use of HTTPS for security. The notification destination must use Secure Sockets Layer (SSL) certificates signed by a trusted certificate authority.
- The notification destination must add both the workspace control plane and data plane outbound IP addresses to its allowlist:
-
For control plane webapp IP addresses, see IP addresses and domains for Databricks services and assets.
-
For data plane outbound IP addresses, see the
ip-ranges.jsonfile that Databricks publishes. Search the file for your workspace's data plane region and allowlist the corresponding outbound IP addresses in your networking stack.Databricks may update the outbound IPs as often as once every 30 days. Updated IPs become active as soon as 60 days after publication, so review the file regularly and update your allowlist before the activation window closes.
To track changes between releases, save successive versions of the JSON file and compare the
timestampSecondsvalue. Filter entries where:- The
servicefield isDatabricks - The
platformfield isaws - The
typefield isoutbound
- The
-
Create a new notification destination
To configure a new notification destination
- Click your username in the top bar of the workspace and select Settings from the dropdown.
- In the Workspace admin section, click the Notifications tab.
- Click the Manage button.
- Click +Add destination.
- Select a destination type. The following destinations are currently supported:
- Slack
- Webhook, with optional OAuth authentication (Beta)
- Microsoft Teams
- PagerDuty
- AWS EventBridge (Beta)
- ServiceNow (Beta)
- Asana (Beta)
- Configure the destination based on the type.
- Click Create.
Webhook payloads
The service that sends a notification defines the webhook payload. For the payloads that jobs send, see HTTP webhook payloads. For the payload that Databricks SQL alerts send after a state change, see SQL alert webhook payload.
Use different credentials for each destination
The configuration of a destination is securely stored encrypted in your Databricks workspace. For destinations that require you to provide a URL or credentials, Databricks recommends using different credentials for each configured destination. This limits the impact if a third-party endpoint is compromised. These include:
- Slack using a webhook: the URL to which the notification is sent.
- Microsoft Teams using a webhook: the URL to which the notification is sent.
- PagerDuty: integration key used to uniquely route notifications to a PagerDuty service.
- Webhook: a username and password for HTTP Basic authentication, or a client ID and client secret for OAuth authentication (Beta).
- Asana: personal access token.
Using different secrets for each configured destination allows you to individually revoke access of individual notification destinations without impacting the operation of all other destinations in your Databricks workspace.
OAuth authentication for webhook destinations
This feature is in Beta. To use it, a workspace admin must turn on Destination Type Generic Webhook Oauth from the Previews page. See Manage workspace-level previews.
Use OAuth 2.0 client credentials to send notifications to an endpoint that requires an access token. Databricks obtains the token from your identity provider and uses it to authenticate the notification request.
Your identity provider must support the client credentials grant. Both the receiving endpoint and the token endpoint must use HTTPS.
To create an OAuth webhook destination:
- In Settings > Notifications, click Manage, then +Add destination.
- Enter a name and select Webhook for Type.
- For Authentication mode, select OAuth.
- Enter the following values:
- URL: The endpoint that receives notifications.
- Client ID and Client secret: The credentials issued by your identity provider.
- Token endpoint URL: The endpoint from which Databricks obtains an access token.
- Scopes and Audience: Enter these optional values if required by your identity provider. Separate multiple scopes with commas.
- Click Create.
The client ID, client secret, and token endpoint URL are required together. OAuth cannot be combined with HTTP Basic authentication.
To verify the configuration, open the destination's three-dot menu and click Test. Databricks obtains an access token and sends a sample notification. If the test fails, check the credentials, scopes, audience, receiver permissions, and network access to both endpoints.
Compliance availability for OAuth webhook destinations
For workspaces with the compliance security profile enabled, region eligibility depends on the selected compliance standards. For region names, see supported regions.
- If one or more standards are selected, open each standard's page from the compliance security profile overview. Find "Destination Type Generic Webhook Oauth" in its regional support table. Your workspace region must be listed for the feature under every selected standard.
- If no compliance standard is selected, no additional standard-specific region restriction applies. Check the workspace's Previews page for Destination Type Generic Webhook Oauth.
AWS EventBridge destination
This feature is in Beta. To use it, a workspace admin must turn on Destination Type Aws Eventbridge from the Previews page. See Manage workspace-level previews.
Workspaces hosted on any supported cloud can send notifications to an Amazon EventBridge event bus in your AWS account. An EventBridge rule routes matching notifications to a target, such as an AWS Lambda function. Notification data leaves Databricks when you send it to EventBridge.
EventBridge requirements
EventBridge setup requires workspace admin access and permission to create or configure the following resources in your AWS account:
- An EventBridge event bus in the region that receives notifications.
- An AWS Identity and Access Management (IAM) role in the same AWS account as the event bus.
- An EventBridge rule and a target that consumes the notifications.
You also need the AWS CLI configured with credentials that have iam:GetRole permission in that AWS account.
Contact your Databricks account team to get the CloudFormation template that creates the IAM role.
The event bus and role must use the commercial AWS partition (arn:aws:). The region in the event bus Amazon Resource Name (ARN) determines where Databricks sends events.
Create an EventBridge destination
-
Create an event bus, or select an existing bus, and copy its ARN.
-
In Databricks, go to Settings > Notifications, click Manage, then +Add destination.
-
Enter a name and select AWS EventBridge for Type.
-
Copy the Databricks principal ARN and External ID displayed in the form.
-
In the AWS account that owns the event bus, open AWS CloudFormation and create a stack from the template that your account team provided. Enter the following template parameters:
DatabricksPrincipalArn: The principal ARN copied from the destination form.ExternalId: The External ID copied from the destination form.EventBusArn: The exact event bus ARN. Don't use wildcards.
Keep the template's trust and permissions policies unchanged. Acknowledge that the stack creates IAM resources, then create the stack. The role grants only
events:PutEventson the specified bus. -
Copy
RoleArnfrom the stack outputs. Run the following command, replacing<role-name>with the final component of that ARN, and send the output to your account team to confirm the trust policy and External ID:Bashaws iam get-role --role-name <role-name> --output json -
After your account team confirms the configuration, return to the destination form and enter the Event bus ARN and Role ARN. Click Create.
The destination appears in the notification destinations list.
Click the destination name to edit its settings, or select Delete from its three-dot menu to remove it. Deleting the destination in Databricks doesn't delete the CloudFormation stack, IAM role, or event bus in your AWS account. Delete those resources separately in AWS if you no longer need them.
The External ID is your Databricks account ID, shared by the workspaces in that account. It isn't a secret. Any workspace admin in the same Databricks account who knows the role and bus ARNs can create a destination that sends to this bus. EventBridge rule filters control which events reach a target. They don't restrict which workspaces can send to the bus.
Databricks uses AWS Security Token Service (STS) to obtain temporary credentials for this role. You don't provide AWS access keys to Databricks.
Route EventBridge notifications to a target
Each EventBridge event has source set to databricks.notifications, a detail-type identifying the notification type, and detail containing the notification payload. The detail.workspace.id field identifies the sending workspace across supported notification types.
-
Create an EventBridge rule on the event bus you configured.
-
Set the event pattern to match the notifications you want to receive. For example, this pattern matches
JOB_RUN_ERROR_V2notifications from one workspace. Replace<workspace-id>with your workspace ID:JSON{"source": ["databricks.notifications"],"detail-type": ["JOB_RUN_ERROR_V2"],"detail": {"workspace": {"id": ["<workspace-id>"]}}}To route all notification types for that workspace, omit
detail-typefrom the pattern. -
Select the rule's target, configure its required permissions, and enable the rule.
EventBridge uses the rule to route matching events to the target. Delivery can produce duplicate notifications. The payload varies by notification type, and the destination doesn't supply a shared deduplication key. For JOB_RUN_ERROR_V2, a target can use detail.workspace.id, detail.run.run_id, detail.run.attempt, and detail-type together to identify repeated notifications.
Slack destination
To set up a webhook-based Slack destination, follow the instructions in Incoming webhooks for Slack. Paste the generated URL into your Databricks notification destination. To use the Databricks Genie app instead, see Slack and Microsoft Teams Genie app destinations.
Slack and Microsoft Teams Genie app destinations
This feature is in Public Preview. To use it, a workspace admin must turn on Destination Type Databricks App Slack or Destination Type Databricks App Teams from the Previews page. See Manage workspace-level previews.
Use the Databricks Genie app to connect your Slack workspace or Microsoft Teams tenant to your Databricks workspace. Then select an available channel as a notification destination.
You can use Genie app destinations with the following features:
- SQL alerts: Slack and Microsoft Teams.
- Unity Catalog access requests: Slack and Microsoft Teams.
- Model Serving endpoint notifications: Slack.
Create a Genie app destination
Before creating a Genie app destination, you must install the Databricks Genie app for Slack or Microsoft Teams, connect it to your Databricks workspace, and add it to the channel that receives notifications. For details on app installation and connection steps, see Slack or Microsoft Teams.
To create a Genie app destination:
-
In Settings > Notifications, click Manage, then +Add destination.
-
For Type, select Slack or Microsoft Teams, then select Using Databricks App.
-
Enter a name for the destination.
-
Select the workspace or tenant and the channel:
- For Slack, select the Slack Workspace and Channel.
- For Microsoft Teams, select the Tenant and Channel. The tenant selector identifies each tenant by the last six characters of its tenant ID.
If no Slack workspaces or Microsoft Teams tenants are connected, click Connect a Slack workspace or Connect a Teams tenant, respectively, and complete the Genie app setup. Return to the destination form and click Refresh. If you just added the Genie app to a channel, click Reload channels to update the channel list.
-
Click Create.
Asana destination
This feature is in Beta. To use it, a workspace admin must turn on Destination Type Asana from the Previews page. See Manage workspace-level previews.
Use an Asana destination to create a task in a project or trigger an existing Asana rule when a notification occurs.
Notifications sent to Asana leave Databricks and are processed and stored according to your Asana access and retention settings. Choose receiving resources with appropriate access controls.
Prepare Asana
Use a dedicated Asana user with access limited to the projects and resources that receive notifications. Create a personal access token (PAT) for that user in the Asana developer console. PATs are long-lived and inherit the user's Asana access.
Prepare the resources for your chosen workflow:
- Task creation: An Asana project and its numeric project ID. Each notification creates a task in that project with a title and notes derived from the notification.
- Rule trigger: An existing Asana rule with an incoming web request trigger, its numeric rule trigger ID, and an existing task for the rule to run against. Asana requires the task even if the rule's actions don't use it.
The rule receives name and notes in action_data for use in its actions. Some notification types also include dedup_key. Retries can trigger the rule again, so design its actions to handle repeated notifications.
Use numeric IDs, not full URLs. For a rule trigger URL containing /rule_triggers/<id>/run, copy only the numeric <id> value. The anchor task ID identifies the existing task. The PAT user must have access to the selected resources.
For project URLs such as https://app.asana.com/1/<workspace-id>/project/<project-id>/list, copy <project-id>. For task URLs such as https://app.asana.com/1/<workspace-id>/project/<project-id>/task/<task-id>, copy <task-id>. Older /0/ URLs use the same project and task IDs.
Create an Asana destination
- In Settings > Notifications, click Manage, then +Add destination.
- Enter a name and select Asana for Type.
- Enter the Personal access token.
- For Workflow, select one of the following:
- Task creation: Enter the numeric Project ID.
- Rule trigger: Enter the numeric Rule trigger ID and Anchor task ID.
- Click Create.
When you change the workflow, project ID, rule trigger ID, or anchor task ID, re-enter the PAT before saving, even if the token hasn't changed. Changing only the destination name doesn't require token re-entry.
PagerDuty destination
To set up a PagerDuty destination:
- In PagerDuty, add an integration to your service and select the Events API V2 integration type.
- Copy the integration key that PagerDuty generates for the integration.
- In Databricks, paste the integration key into the PagerDuty notification destination.
ServiceNow destination
This feature is in Beta. To use it, a workspace admin must turn on Destination Type ServiceNow from the Previews page. See Manage workspace-level previews.
Send notifications to a ServiceNow table using the Table API and OAuth 2.0 client credentials.
ServiceNow requirements
Before you create the destination, ask your ServiceNow administrator to prepare the following:
- A dedicated integration user with permission to create records in your table. Limit the user's roles and table access to the permissions required for the integration.
- An OAuth application that supports the client credentials grant, with a client ID, client secret, and HTTPS token endpoint. Configure the application as follows:
- Set OAuth Application User to the dedicated integration user. If the field is hidden, add it to the form layout.
- Allow Table API calls without an explicitly requested OAuth scope. Databricks does not request a scope.
- A table accessible through an HTTPS Table API URL, such as
https://<instance>.service-now.com/api/now/table/<table_name>. - Table columns that match the names in the following table. Use the Application or Global names that correspond to your table's scope. The payload column must accommodate the notification's JSON content as a string.
Application scope column | Global scope column | Content |
|---|---|---|
|
| Notification event type. |
|
| Event identifier. This can be empty for some events. |
|
| Event description. |
|
| Event timestamp. This can be empty for some events. |
|
| Event-specific JSON serialized as a string. |
The URL and Token endpoint URL values must not contain credentials or sensitive query parameters. Notification data leaves Databricks and is sent to your ServiceNow instance, which can be outside your workspace's region. Configure access and retention in ServiceNow to meet your organization's requirements.
Create a ServiceNow destination
- In Settings > Notifications, click Manage, then +Add destination.
- Enter a name and select ServiceNow for Type.
- Enter the following values:
- URL: The ServiceNow Table API endpoint for your table.
- Client ID and Client secret: The credentials issued by your ServiceNow OAuth application.
- Token endpoint URL: Your ServiceNow OAuth application's HTTPS token endpoint, such as
https://<instance>.service-now.com/oauth_token.do. - Table scope: Select Application or Global to match the column names in your table.
- Click Create.
After you create the destination, send a notification through your normal workflow and confirm that a new record appears in your ServiceNow table.
To edit the destination or rotate its credentials, click the destination name, update the fields, and click Save. Re-enter the client secret when changing URL or Token endpoint URL. To delete the destination, open its three-dot menu, click Delete, and confirm deletion.
Add a webhook to a job
After configuring destinations, you can access them in a job's Job notifications settings. For more information, see Add notifications on a job.
Each job supports up to three system destinations per event type. When configuring job notifications, email addresses must be entered manually.
Limitations
Notification destinations currently has the following limitations:
- Email notification destinations have a 1,300-character limit on the recipient address length.
- For custom notification bodies, non-email destinations such as Slack and Microsoft Teams do not support HTML formatting. Some notification destinations support markdown.
- For jobs, email destinations must be manually set in the job's settings.