Connect to data sources
This article provides opinionated recommendations for how administrators and other power users can configure connections between Databricks and data sources. If you are trying to determine whether you have access to read data from an external system, start by reviewing the data that you have access to in your workspace. See Discover data.
For information about connecting to external services that are not necessarily data sources, see Manage access to external cloud services using service credentials.
You can connect your Databricks account to data sources such as cloud object storage, relational database management systems, streaming data services, and enterprise platforms such as CRMs. The specific privileges required to configure connections depends on the data source, how permissions in your Databricks workspace are configured, the required permissions for interacting with data in the source, your data governance model, and your preferred method for connecting.
Most methods require elevated privileges on both the data source and the Databricks workspace to configure the necessary permissions to integrate systems. Users without these permissions should request help. See Request access to data sources.
Configure object storage connections
Cloud object storage provides the basis for storing most data on Databricks. To learn more about cloud object storage and where Databricks stores data, see Where does Databricks write data?.
Databricks recommends using Unity Catalog to configure access to cloud object storage. Unity Catalog provides data governance for both structured and unstructured data in cloud object storage. See Connect to cloud object storage and services using Unity Catalog.
Customers who don't use Unity Catalog must configure connections using legacy methods. See Configure access to cloud object storage for Databricks.
To configure networking to cloud object storage, see Networking.
Configure connections to external systems
Databricks recommends several options for configuring connections to external data systems depending on your needs. The following table provides a high-level overview of these options:
Option | Description |
|---|---|
Query federation connectors | Lakehouse Federation provides read-only access to data in enterprise data systems. Connections are configured through Unity Catalog at the catalog or schema level, syncing multiple tables with a single configuration. See What is Lakehouse Federation?. |
Managed ingestion connectors | Lakeflow Connect allows admin users to create a connection and a managed ingestion pipeline at the same time in the data ingestion UI. See Managed connectors in Lakeflow Connect. If the users who will create pipelines are non-admin users or plan to use Databricks APIs, Databricks SDKs, the Databricks CLI, or Databricks Asset Bundles, an admin must first create the connection in Catalog Explorer. These interfaces require that users specify an existing connection when they create a pipeline. See Connect to managed ingestion sources. |
Streaming connectors | Databricks provides optimized connectors for many streaming data systems. For all streaming data sources, you must generate credentials that provide access and load these credentials into Databricks. Databricks recommends storing credentials using secrets, because you can use secrets for all configuration options and in all access modes. All data connectors for streaming sources support passing credentials using options when you define streaming queries. See Standard connectors in Lakeflow Connect. |
Third-party integrations | Use third-party tools to connect to external data sources and automate ingesting data to the lakehouse. Some solutions also include reverse ETL and direct access to lakehouse data from external systems. See What is Databricks Partner Connect?. |
Drivers | Databricks includes drivers for external data systems in each Databricks Runtime. You can optionally install third-party drivers to access data in other systems. You must configure connections for each table. Some drivers include write access. See Connect to external systems. |
JDBC | Several included drivers for external systems build upon native JDBC support, and the JDBC option provides extensible options for configuring connections to other systems. You must configure connections for each table. See Query databases using JDBC. |
Request access to data sources
In many organizations, most users do not have sufficient privileges on either Databricks or external data sources to configure data connections.
Your organization might have already configured access to a data source using one of the patterns described in the articles linked from this page. If your organization has a well-defined process for requesting access to data, Databricks recommends following that process.
If you're uncertain how to gain access to a data source, this procedure might help you:
- Use Catalog Explorer to view the tables and volumes that you can access. See What is Catalog Explorer?.
- Ask your teammates or managers about the data sources that they can access.
- Most organizations use groups synced from their identity provider (for example: Okta or Microsoft Entra ID) to manage permissions for workspace users. If other members of your team can access data sources that you need access to, have a workspace admin add you to the correct group to grant you access.
- If a particular table, volume, or data source was configured by a co-worker, that individual should have permissions to grant you access to the data.
- Some organizations configure data access permissions through settings on compute clusters and SQL warehouses.
- Access to data sources can vary by compute.
- You can view the compute creator on the Compute tab. Reach out to the creator to ask about data sources that should be accessible.