Cluster configuration for Databricks Connect

Note

This article covers Databricks Connect for Databricks Runtime 13.0 and above.

This article lists configuration requirements and settings for Databricks clusters before Databricks Connect can connect to them. This information applies to the Python and Scala version of Databricks Connect unless stated otherwise.

Databricks Connect enables you to connect popular IDEs such as Visual Studio Code, PyCharm, RStudio Desktop, IntelliJ IDEA, notebook servers, and other custom applications to Databricks clusters. See What is Databricks Connect?.

Requirements

  • You have a Databricks workspace and its corresponding account that are enabled for Unity Catalog. See Set up and manage Unity Catalog and Enable a workspace for Unity Catalog.

  • A Databricks cluster with Databricks Runtime 13.3 LTS or higher installed.

  • The cluster must use a cluster access mode of Assigned or Shared. See Access modes.

  • The version of Databricks Connect used must be at or lower than the Databricks Runtime version running on the cluster.

Programmatic validation

In Databricks Connect 14.3 and above, DatabricksSession.builder introduces validateSession, which runs a series of validations to ensure that the preceding requirements are met.

In Databricks Connect for Python, the databricks-connect binary has a test subcommand that performs the same set of validations.

This command should be run on the terminal with an active Python environment which includes Databricks Connect, and with the set of default credentials configured. To configure these credentials, see Configure connection properties.

databricks-connect test

The command will fail with a non-zero exit code and an appropriate message when any of the requirements are not met.

Disabling Databricks Connect

Databricks Connect (and the underlying Spark Connect) services can be disabled on any given cluster.

To disable the Databricks Connect service, set the following Spark configuration on the cluster.

spark.databricks.service.server.enabled false