Connect to Fivetran

Fivetran automated data integration adapts as schemas and APIs change, ensuring reliable data access and simplified analysis with ready-to-query schemas.

You can integrate your Databricks SQL warehouses (formerly Databricks SQL endpoints) and Databricks clusters with Fivetran. The Fivetran integration with Databricks helps you centralize data from disparate data sources into Delta Lake.

Connect to Fivetran using Partner Connect

This section describes how to connect to Fivetran using Partner Connect. Each user creates their own connection.

Note

The per-user connection experience is in Public Preview. You can disable the ability to create per-user connections by contacting your Databricks account team.

Before you connect using Partner Connect

Before you connect to Fivetran using Partner Connect, make sure you have the following:

  • The workspace admin user role or the following permissions:

    • The CAN USE permission for a SQL warehouse

    • The CAN USE permission for token usage

  • For data managed by Unity Catalog, the following metastore object privileges for the catalog you want Fivetran to write to:

    • USE CATALOG and`CREATE SCHEMA` on the catalog.

    • (Optional) To specify a destination location, CREATE EXTERNAL TABLE on the external location and access to data in cloud object storage.

  • For data managed by the legacy Hive metastore, the following metastore object privileges for the catalog you want Fivetran to write to:

    • USAGE and CREATE on the catalog

    • (Optional) To specify a destination location, access to data in cloud object storage.

Privileges for Unity Catalog metastore objects can be granted by a metastore admin, the owner of the object, or the owner of the catalog or schema that contains the object. For more information, see Unity Catalog privileges and securable objects.

Privileges for legacy Hive metastore objects can be granted by a workspace admin or the owner of the object. For more information, see Hive metastore privileges and securable objects (legacy).

Access to data in cloud object storage must be configured by a workspace admin with sufficient permissions in the cloud object storage account. For more information, see Enable data access configuration.

Partner Connect steps

To connect your Databricks workspace to Fivetran using Partner Connect, do the following:

  1. In the sidebar, click Partner Connect.

  2. Click the Fivetran tile.

    The steps in this section create a user-level Fivetran trial account. To sign in to an existing workspace-level Fivetran trial account, click Use existing connection, complete the on-screen instructions to sign in to Fivetran, and skip the rest of the steps in this article.

  3. Select a SQL warehouse. If the SQL warehouse is stopped, click Start.

  4. If your workspace is enabled for Unity Catalog, select a catalog for Fivetran to write to, then click Next.

    Partner Connect generates a Databricks personal access token that is associated with your user.

  5. Click Connect to Fivetran.

    A new tab opens in your web browser that displays the Fivetran website.

  6. Complete the on-screen instructions on the Fivetran website to create your trial partner account.

Connect to Fivetran manually

For an overview of the manual connection procedure, watch this YouTube video (2 minutes).

Note

To connect a SQL warehouse with Fivetran faster, use Partner Connect.

Before you connect manually

Before you connect to Fivetran manually, you must have the following:

  • A cluster or SQL warehouse in your Databricks workspace.

  • The connection details for your cluster or SQL warehouse, specifically the Server Hostname, Port, and HTTP Path values.

  • A Databricks personal access token. To create a personal access token, do the following:

    1. In your Databricks workspace, click your Databricks username in the top bar, and then select Settings from the drop down.

    2. Click Developer.

    3. Next to Access tokens, click Manage.

    4. Click Generate new token.

    5. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).

    6. Click Generate.

    7. Copy the displayed token to a secure location, and then click Done.

    Note

    Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token. If you lose the copied token, or you believe that the token has been compromised, Databricks strongly recommends that you immediately delete that token from your workspace by clicking the trash can (Revoke) icon next to the token on the Access tokens page.

    If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following topics:

    Note

    As a security best practice when you authenticate with automated tools, systems, scripts, and apps, Databricks recommends that you use OAuth tokens.

    If you use personal access token authentication, Databricks recommends using personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service principal.

Tip

If the Fivetran tile in Partner Connect in your workspace has a check mark icon inside of it, you can get the connection details for the connected SQL warehouse by clicking the tile and then expanding Connection details. The Personal access token is hidden; you must create a replacement personal access token and enter that new token instead when Fivetran asks you for it.

Manual steps

To connect to Fivetran manually, do the following:

  1. Sign in to your Fivetran account, or create a new Fivetran account, at https://fivetran.com/login.

    Important

    If you sign in to your organization’s Fivetran account, a Choose Destination page may display, listing one or more existing destination entries with the Databricks logo. These entries might contain connection details for compute resources in workspaces that are separate from yours. If you still want to reuse one of these connections, and you trust the compute resource and have access to it, choose that destination and then skip ahead to next steps. Otherwise, choose any available destination to get past this page.

  2. In your Dashboard page in Fivetran, click the Destinations tab.

  3. Click Add Destination.

  4. Enter a Destination name and click Add.

  5. On the Fivetran is modern ELT page, click Set up a connector.

  6. Click a data source, and then click Next.

  7. Follow the on-screen instructions in the Setup Guide in Fivetran to finish setting up the connector.

  8. Click Save & Test.

  9. After the test succeeds, click Continue.

  10. On the Select your data’s destination page, click Databricks on AWS.

  11. Click Continue Setup.

  12. Complete the on-screen instructions in Fivetran to enter the connection details for your existing Databricks compute resource, specifically the Server Hostname and HTTP Path field values, and the token that you generated earlier.

  13. Click Save & Test.

  14. After the test succeeeds, click Continue.

  15. Continue to next steps.

Additional resources

Explore one or more of the following resources on the Fivetran website: