This feature is available only if your account is on the E2 version of the Databricks platform. Secure cluster connectivity is enabled for accounts on the E2 platform by default as of September 1, 2020.
With secure cluster connectivity, customer VPCs have no open ports and Databricks Runtime workers have no public IP addresses.
- At a network level, each cluster initiates a connection to the control plane secure cluster connectivity relay (proxy) during cluster creation. The cluster establishes this connection using port 443 (HTTPS) and a different IP address than is used for the Web application and REST API.
- Actions that the control plane logically initiates, such as starting new Databricks Runtime jobs or performing cluster administration, are sent as requests to the cluster through this reverse tunnel.
- The data plane (the customer VPC) does not have open ports and Databricks Runtime workers do not have public IP addresses.
- Easy network administration — Less complexity because there is no need for port configuration on security groups or configuring network peering.
- Easier approval — Because of better security and simpler network administration, it is easier for Information Security teams to approve Databricks as a PaaS provider.
To use secure cluster connectivity for a workspace, you must create a new workspace using the Account API. You cannot add secure cluster connectivity to an existing workspace.