Enhanced security monitoring

Databricks enhanced security monitoring provides an enhanced hardened disk image and additional security monitoring agents that generate log rows that you can review.

To review the new log rows, set up audit log delivery. For additional information about audit logs, see Audit log reference.

Which compute resources get enhanced monitoring

The security enhancements apply only to compute resources in the classic compute plane, such as clusters and non-serverless SQL warehouses.

Serverless SQL warehouses do not have extra monitoring when enhanced security monitoring is enabled.

Features and technical controls

  • Enhanced disk image (a CIS-hardened Ubuntu Advantage AMI)

  • Antivirus monitoring agent that generate logs that you can review.

  • File integrity monitoring agent that generate logs that you can review.

Requirements

  • Your Databricks workspace is on the E2 version of the platform.

  • Your Databricks workspace is on the Enterprise pricing tier.

  • Your Databricks account must include the Enhanced Security and Compliance add-on. For details, see the pricing page.

Enable Databricks enhanced security monitoring

To enable the enhanced security monitoring directly on a workspace, see Enable features on a workspace.

You can also set an account-level default for new workspaces to enable enhanced security monitoring initially. Alternatively, you can set an account-level default to enable the compliance security profile, which automatically enables enhanced security monitoring. See Set account-level defaults for new workspaces.

Updates may take up to six hours to propagate to all environments and to downstream systems like billing. Workloads that are actively running continue with the settings that were active at the time of starting the cluster or other compute resource, and new settings will start applying the next time these workloads are started. This means that if a change is made late in the day or on the last day of the month, you might still see usage reported with the old settings early the next day or month.

  1. Restart your compute resources.

Enable enhanced security monitoring automatically for new workspaces

Account admins can set account-level defaults for new workspaces for enhanced security monitoring. See Set account-level defaults for new workspaces.

Disk image with enhanced hardening

While Databricks enhanced security monitoring is enabled, Databricks compute resources (cluster worker images) in your classic compute plane use an enhanced hardened operating system image based on Ubuntu Advantage.

Ubuntu Advantage is a package of enterprise security and support for open source infrastructure and applications that includes the following:

Monitoring agents in Databricks compute plane images

While Databricks enhanced security monitoring is enabled, there are additional security monitoring agents, including two agents that are pre-installed in the enhanced compute plane image. You cannot disable the monitoring agents that are in the enhanced compute plane disk image.

Monitoring agent

Location

Description

How to get output

File integrity monitoring

Enhanced compute plane image

Monitors for file integrity and security boundary violations. This monitor agent runs on the worker VM in your cluster.

Enable the audit log system table and review logs for new rows.

Antivirus and malware detection

Enhanced compute plane image

Scans the filesystem for viruses daily. This monitor agent runs on the VMs in your compute resources such as clusters and pro or classic SQL warehouses. The antivirus and malware detection agent scans the entire host OS filesystem and the Databricks Runtime container filesystem. Anything outside the cluster VMs is outside of its scanning scope.

Enable the audit log system table and review logs for new rows.

Vulnerability scanning

Scanning happens in representative images in the Databricks environments.

Scans the container host (VM) for certain known vulnerabilities and CVEs.

Request scan reports on the image from your Databricks account team.

To get the latest versions of monitoring agents, you can restart your clusters. If your workspace uses automatic cluster update, by default clusters restart if needed during the scheduled maintenance windows. If the compliance security profile is enabled on a workspace, automatic cluster update is permanently enabled on that workspace.

File integrity monitoring

The enhanced compute plane image includes a file integrity monitoring service that provides runtime visibility and threat detection for compute resources (cluster workers) in the classic compute plane in your workspace.

The file integrity monitor output is generated within your audit logs, which you can access with system tables (Public Preview). For the JSON schema for new auditable events that are specific to file integrity monitoring, see File integrity monitoring events.

Important

It is your responsibility to review these logs. Databricks may, in its sole discretion, review these logs but does not make a commitment to do so. If the agent detects a malicious activity, it is your responsibility to triage these events and open a support ticket with Databricks if the resolution or remediation requires an action by Databricks. Databricks may take action on the basis of these logs, including suspending or terminating the resources, but does not make any commitment to do so.

Antivirus and malware detection

The enhanced compute plane image includes an antivirus engine for detecting trojans, viruses, malware, and other malicious threats. The antivirus monitor scans the entire host OS filesystem and the Databricks Runtime container filesystem. Anything outside the cluster VMs is outside of its scanning scope.

The antivirus monitor output is generated within audit logs, which you can access with system tables (Public Preview). For the JSON schema for new auditable events that are specific to antivirus monitoring, see Antivirus monitoring events.

When a new virtual machine image is built, updated signature files are included within it.

Important

It is your responsibility to review these logs. Databricks may, in its sole discretion, review these logs but does not make a commitment to do so. If the agent detects a malicious activity, it is your responsibility to triage these events and open a support ticket with Databricks if the resolution or remediation requires an action by Databricks. Databricks may take action on the basis of these logs, including suspending or terminating the resources, but does not make any commitment to do so.

When a new AMI image is built, updated signature files are included within the new AMI image.

Vulnerability scanning

A vulnerability monitor agent performs vulnerability scans of the container host (VM) for certain known CVEs.

Important

The scanning happens in representative images in the Databricks environments.

You can request the vulnerability scan reports from your Databricks account team.

When vulnerabilities are found with this agent, Databricks tracks them against its Vulnerability Management SLA and releases an updated image when available.

Management and upgrade of monitoring agents

The additional monitoring agents that are on the disk images used for the compute resources in the classic compute plane are part of the standard Databricks process for upgrading systems:

  • The classic compute plane base disk image (AMI) is owned, managed, and patched by Databricks.

  • Databricks delivers and applies security patches by releasing new AMI disk images. The delivery schedule depends on new functionality and the SLA for discovered vulnerabilities. Typical delivery is every two to four weeks.

  • The base operating system for the compute plane is Ubuntu Advantage.

  • Databricks clusters and pro or classic SQL warehouses are ephemeral by default. Upon launch, clusters and pro or classic SQL warehouses use the latest available base image. Older versions that may have security vulnerabilities are unavailable for new clusters.

    • You are responsible for restarting clusters (using the UI or API) regularly to ensure they use the latest patched host VM images.

    • Databricks can, upon request, share a Databricks notebook to identify your workspace’s running clusters and hosts older than a specified number of days and optionally, restart a cluster.

Monitor agent termination

If a monitor agent on the worker VM is found to be not running due to crash or other termination, the system will attempt to restart the agent.

Data retention policy for monitor agent data

Monitoring logs are sent to the audit log system table or your own Amazon S3 bucket if you configured audit log delivery. Retention, ingestion, and analysis of these logs is your responsibility.

Vulnerability scanning reports and logs are retained for at least one year by Databricks. You can request the vulnerability reports from your Databricks account team.