Configure enhanced security and compliance settings

This page describes how to configure enhanced security and compliance settings on your Databricks workspace.

important

• Enabling the compliance security profile or adding compliance standards to a workspace is intended to be a permanent change.
• You cannot remove the compliance profile or individual standards from a workspace if it has ever processed regulated data. To revert, you must delete the workspace and create a new one without the profile or with a different standard. For help, contact Databricks support.
• Before processing PHI, make sure you have a BAA agreement in place with Databricks.

Requirements

• Your Databricks account must include the Enhanced Security and Compliance add-on. For details, see the pricing page.
• Your Databricks workspace is on the Enterprise pricing tier.

Compliance security profile requirements

• Single sign-on (SSO) authentication is configured for the workspace.

• Your Databricks workspace storage bucket cannot have a period character (.) in its name, such as my-bucket-1.0. If an existing workspace storage bucket has a period character in the name, contact your Databricks account team before enabling the compliance security profile.

• You must add the required network ports. The network ports you need depend on whether PrivateLink is enabled for private connectivity to the classic compute plane.

  • PrivateLink enabled: Update your network security group to allow bidirectional access on port 2443 for FIPS encryption. See Create VPC.
  • PrivateLink not enabled: Allow outbound access on port 2443 to support FIPS endpoints. See Security groups.

• If your workspace is in the us-east-1, us-east-2, us-west-1, us-west-2, or ca-central-1 region and restricts outbound traffic, you must allow access to additional S3 endpoints to support FIPS. This applies only to S3. AWS does not provide FIPS endpoints for STS or Kinesis.

  • Allow outbound traffic to s3.<region>.amazonaws.com and s3-fips.<region>.amazonaws.com. For example: s3.us-east-1.amazonaws.com and s3-fips.us-east-1.amazonaws.com.

Supported instance types

Instance types are limited to those that provide both hardware-implemented network encryption between cluster nodes and encryption at rest for local disks. Compliance standards that enforce FIPS mode also do not support Graviton instances and AWS GovCloud does not support fleet instances.

• The supported instance types for HIPAA, PCI-DSS, K-FSI, and UK Cyber Essentials Plus are:

  • General purpose: M-fleet, Md-fleet, M5dn, M5n, M5zn, M7g, M7gd, M6i, M7i, M6id, M6in, M6idn
  • Compute optimized: C-fleet, C5a, C5ad, C5n, C6gn, C7g, C7gd, C7gn, C6i, C6id, C7i, C6in
  • Memory optimized: R-fleet, Rd-fleet, R7g, R7gd, R6i, R7i, R7iz, R6id, R6in, R6idn
  • Storage optimized: D3, D3en, P3dn, R5dn, R5n, I4i, I3en
  • Accelerated computing: G4dn, G5, P4d, P4de, P5

• The supported instance types for FedRAMP Moderate, IRAP, and CCCS Medium (Protected B) are:

  • General purpose: M-fleet, Md-fleet, M5dn, M5n, M5zn, M6i, M7i, M6id, M6in, M6idn
  • Compute optimized: C-fleet, C5a, C5ad, C5n, C6i, C6id, C7i, C6in
  • Memory optimized: R-fleet, Rd-fleet, R6i, R7i, R7iz, R6id, R6in, R6idn
  • Storage optimized: D3, D3en, P3dn, R5dn, R5n, I4i, I3en
  • Accelerated computing: G4dn, G5, P4d, P4de, P5

• The supported instance types for DoD IL5 and FedRAMP High are:

  • General purpose: M5dn, M5n, M5zn, M6i, M7i, M6id, M6in, M6idn
  • Compute optimized: C5a, C5ad, C5n, C6i, C6id, C7i, C6in
  • Memory optimized: R6i, R7i, R7iz, R6id, R6in, R6idn
  • Storage optimized: D3, D3en, P3dn, R5dn, R5n, I4i, I3en
  • Accelerated computing: G4dn, G5, P4d, P4de, P5

• Ensure that sensitive information is never entered in customer-defined input fields, such as workspace names, cluster names, tags, and job names.

note

Databricks Assistant is disabled by default on workspaces that have enabled the compliance security profile. Workspace admins can enable it by following the instructions For an account: Disable or enable partner-powered AI features.

Enable on an existing workspace

Account admins can enable the compliance security profile, add compliance standards, enable enhanced security monitoring, and enable automatic cluster update on a workspace.

Verify networking requirements for the compliance security profile

If you are enabling the compliance security profile, after updating network access according to the requirements above, verify connectivity and configurations with the following tests:

• DBFS access: In a notebook attached to a test cluster, run:

  Bash

  %fs ls /
  %sh ls /dbfs

  Confirm that both commands return a file listing without errors.

• Control plane access: Use nc to test connectivity to your region’s webapp domain on port 443. For example:

  Bash

  %sh nc -zv oregon.cloud.databricks.com 443

• SCC relay access:

  • If PrivateLink is not enabled, use port 2443:

    Bash

    %sh nc -zv <scc-relay-domain-name> 2443

  • If PrivateLink is enabled, use port 6666:

    Bash

    %sh nc -zv <scc-relay-for-privatelink-domain-name> 6666

• S3 FIPS endpoint access (only for us-east-1, us-east-2, us-west-1, us-west-2, or ca-central-1):

  Bash

  %sh nc -zv <bucket-name>.s3-fips.<region>.amazonaws.com 443

• Spark configuration: Confirm that Spark is using the correct endpoints:

  Python

  >>> spark.conf.get("fs.s3a.stsAssumeRole.stsEndpoint")
  
  "sts.us-west-1.amazonaws.com"
  
  >>> spark.conf.get("fs.s3a.endpoint")
  
  "s3-fips.us-west-2.amazonaws.com"
  
  

Enable enhanced security and compliance settings

1. Check for running compute. Make sure no important jobs are running. Enabling the compliance security profile or automatic cluster update will automatically restart clusters.

2. As an account admin, go to the account console.

3. Click Workspaces.

4. Click on your workspace's name.

5. Click Security and compliance.

6. To enable the compliance security profile, next to Compliance security profile, click Enable.

   In the Compliance security profile dialog, optionally select compliance standards and click Save.

   

   important

   You must have a BAA agreement with Databricks before processing any PHI data. Enabling HIPAA in the compliance security profile does not replace this requirement.

7. To enable enhanced security monitoring, select the checkbox Enhanced security monitoring.

   If you enable the compliance security profile, enhanced security monitoring is automatically enabled.

8. To enable automatic cluster update, select or unselect the checkbox Automatic cluster update.

   If you enable the compliance security profile, automatic cluster update is automatically enabled.

Set account-level defaults for all new workspaces

Account admins can configure settings for security profile (with compliance standards) or enhanced security monitoring at an account level to apply to all new workspaces. When you enable the compliance security profile as a default for new workspaces, enhanced security monitoring and automatic cluster update are also enabled for new workspaces.

1. As an account admin, go to the account console.

2. In the sidebar, click Settings.

3. Click the Security and compliance tab.

4. In the sidebar, click Enhanced Security and Compliance Settings.

5. To enable the compliance security profile, next to Compliance security profile, click Configure.

   In the Compliance security profile for new workspaces dialog, select Enabled, select one or compliance standards or select None and click Save.

6. To enable enhanced security monitoring, select the checkbox Enhanced security monitoring for new workspaces.

Enable compliance settings during workspace deployment

Account admins can enable the compliance security profile, add compliance standards, and enable enhanced security monitoring during workspace creation. New workspaces must adhere to configured account-level defaults.

1. Follow the instructions to create a workspace in Create a workspace with custom AWS configurations. Ensure you choose a region that supports your compliance standard.
2. Click Advanced configurations.
3. To enable the compliance security profile, select the checkbox Compliance security profile.
4. To enable enhanced security monitoring, select the checkbox Enhanced security monitoring.
5. In Compliance standard, select one or more compliance standards or select None.
6. Click Next and Create workspace.

Confirm that the compliance security profile is enabled for a workspace

You can confirm a workspace is using the compliance security profile in the Security and compliance tab on the workspace page in the account console.

The workspace also has a shield logo displayed in the workspace UI. A shield logo appears in the top-right of the page, to the right of the workspace name. Click the workspace name to see a list of the workspaces that you have access to. The workspaces that enable the compliance security profile have a shield icon.

If the shield icons are missing for a workspace with the compliance security profile enabled, contact your Databricks account team.