Configure Azure storage firewalls to allow access from serverless SQL warehouses

Preview

Serverless SQL warehouses are available in Public Preview.

If you already use or plan to use an Azure Storage firewall, you must configure the firewall to allow access from serverless SQL warehouses, which are compute resources that run in the Azure subscription for Databricks not your Azure subscription.

You must allow network access based on service endpoints from all of the subnets where serverless SQL warehouses might reach each Azure storage account. You must have WRITE access to your storage account network rules to make these configuration changes. There are multiple subnets in each region to ensure availability.

  1. Get the list of subnets for your workspace region. See Databricks SQL Serverless subnets for each region.

    Although the subnet list for your region might add additional subnets later, this list is stable over time for an individual workspace. Databricks uses your workspace’s creation date to avoid access from newer subnets that were recently added and might not be configured for firewall access.

    Warning

    Each time you create a new workspace, it is critical that you return to this page to get the latest subnet list. Do not assume the list is the same as for the last workspace that you created in that region. Failure to get the latest subnet list for new workspaces might result in runtime failures when attempting data access from serverless SQL warehouse subnets that were not allowlisted.

  2. Add one Azure storage account network rule for each subnet. You can do this using the Azure Portal, Azure CLI, Powershell, Terraform, or other automation tools. The following example uses the Azure CLI:

    • Replace <sub> with the name of your Azure subscription for the storage account.

    • Replace <res> with the resource group of customer’s storage account.

    • Replace <account> with the name of your storage account

    • Replace <subnet> with the ARM resource ID (resourceId) of the serverless SQL warehouse subnet.

    az storage account network-rule add --subscription "<sub>" \
       --resource-group "<res>" --account-name "<account>" --subnet "<subnet>"
    

    After running all the commands, you can use the Azure portal to view your storage account and confirm that there is an entry in the “Virtual Networks” table that represents the new subnet.

    Tip

    Do not worry about the “Insufficient permissions” reference in the endpoint status column or the warning below the network list. That indicates only that you do not have permissions to read the Databricks subnets but it does not interfere with the ability for that subnet to contact your Azure storage.

    Example new entries in Virtual Networks list
  3. Repeat this command once each for every subnet.

  4. If you have multiple storage accounts, repeat the previous steps for each additional storage account that needs access to that storage for that same region. If you need to configure access for other workspace regions, be sure to get the latest list for that region.

For example, suppose that you want to allow access to two storage accounts in a region that had 12 subnets listed in Databricks SQL Serverless subnets for each region for that region. That would mean that you will run 12 commands (one for each subnet) for each storage account, with 24 commands in total.

Depending on what tools you use, you can automate this. For example, using the bash shell for Unix, Linux, or macOS. Here is an example that uses two subnets in a list that you can use with a loop to run the command for each subnet. Adapt this example as needed for your use case and the complete set of subnets for your region. In this example, mystorage-rg is the resource group and myaccount is the storage account.

#!/bin/bash
SUBNETS=(/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos2/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet /subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet)
for SUBNET in ${SUBNETS[@]}
do
  az storage account network-rule add --subscription 9999999-1ff3-43f4-b91e-d0ceb97111111 --resource-group mystorage-rg --account-name myaccount --subnet ${SUBNET}
done

Databricks SQL Serverless subnets for each region

Refer to the following sections for Databricks SQL Serverless subnets for each region. Each line is a subnet that might need to access your Azure storage account for your workspace region. For example, if there are 12 lines listed for your workspace region, you must add one network rule for each of these 12 regional subnets for each storage account.

East US

/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos2/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastusc3-nephos7/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos1-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos2-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos3-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos4-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos5-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus-nephos6-xr/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet

West US

/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos7/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westus-nephos8/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet

West Europe

/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos2/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeurope-nephos7/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos1/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos2/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-westeuropec2-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet

East US 2

/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos7/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos8/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2-nephos9/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos3/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos4/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos5/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos6/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos7/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet
/subscriptions/8453a5d5-9e9e-40c7-87a4-0ab4cc197f48/resourceGroups/prod-azure-eastus2c2-nephos8/providers/Microsoft.Network/virtualNetworks/kaas-vnet/subnets/worker-subnet