Cluster configuration for Databricks Connect
Note
This article covers Databricks Connect for Databricks Runtime 13.0 and above.
This article lists configuration requirements and settings for Databricks clusters before Databricks Connect can connect to them. This information applies to the Python and Scala version of Databricks Connect unless stated otherwise.
Databricks Connect enables you to connect popular IDEs such as Visual Studio Code, PyCharm, RStudio Desktop, IntelliJ IDEA, notebook servers, and other custom applications to Databricks clusters. See What is Databricks Connect?.
Requirements
You have a Databricks workspace and its corresponding account that are enabled for Unity Catalog. See Set up and manage Unity Catalog and Enable a workspace for Unity Catalog.
A Databricks cluster with Databricks Runtime 13.3 LTS or higher installed.
The cluster must use a cluster access mode of Assigned or Shared. See Access modes.
The version of Databricks Connect used must be at or lower than the Databricks Runtime version running on the cluster.
Programmatic validation
In Databricks Connect 14.3 and above, DatabricksSession.builder
introduces validateSession
, which runs a series of validations to ensure that the preceding requirements are met.
In Databricks Connect for Python, the databricks-connect
binary has a test
subcommand that performs the same set of validations.
This command should be run on the terminal with an active Python environment which includes Databricks Connect, and with the set of default credentials configured. To configure these credentials, see Configure connection properties.
databricks-connect test
The command will fail with a non-zero exit code and an appropriate message when any of the requirements are not met.
Disabling Databricks Connect
Databricks Connect (and the underlying Spark Connect) services can be disabled on any given cluster.
To disable the Databricks Connect service, set the following Spark configuration on the cluster.
spark.databricks.service.server.enabled false
Next steps
For Python, see Install Databricks Connect for Python.
For R, see Databricks Connect for R.
For Scala, see Install Databricks Connect for Scala.