This article provides syntax examples of using the Databricks connector to connect to another Databricks workspace. This connector leverages the Databricks JDBC driver, which is included in Databricks Runtime 13.0 and greater.
For most data sharing operations, Databricks recommends Delta Sharing. See Share data securely using Delta Sharing. You may also prefer Lakehouse Federation for managing queries on data in other Databricks workspaces. See Run queries using Lakehouse Federation.
The Databricks Spark connector allows you to connect to compute resources configured in another Databricks workspace and return results to your current Databricks workspace. You must have access to active compute on both workspaces for queries to succeed.
The JDBC driver is registered for
jdbc:databricks:// URLs. You must configure and use a personal access token that grants you permissions on the workspace resources being accessed remotely. See the Token management API.
If you have a Databricks JDBC library attached to your cluster, the library version attached your cluster is used instead of the version included in Databricks Runtime.
You can specify the format
databricks to use the Databricks Spark connector when you’re reading data, as in the following example:
df = (spark.read .format("databricks") .option("host", "<host-name>.cloud.databricks.com") .option("httpPath", "/sql/1.0/warehouses/<warehouse-id>") .option("personalAccessToken", "<auth-token>") .option("dbtable", "<table-name>") .load() )
You can register an external table in a Databricks workspace linked to a separate Databricks workspace.
The following example demonstrates this syntax, using the
secret function to get credentials stored with Databricks secrets:
For more on Databricks secrets, see secret function.
CREATE TABLE databricks_external_table USING databricks OPTIONS ( host '<host-name>.cloud.databricks.com', httpPath '/sql/1.0/warehouses/<warehouse-id>', personalAccessToken secret('<scope>', '<token>'), dbtable '<table-name>' );