Prophecy helps teams be successful and productive on Apache Spark and Apache Airflow with low-code development, scheduling, and metadata.
You can integrate your Databricks clusters with Prophecy.
For a general overview and demonstration of Prophecy, watch the following YouTube video (26 minutes).
Prophecy does not integrate with Databricks SQL endpoints.
If you already have a Prophecy account, Databricks recommends that you skip ahead to Connect to Prophecy manually instead. This is because the new connection experience in Partner Connect is optimized for new Prophecy accounts.
For an overview of the following procedure, watch this YouTube video (3 minutes).
Make sure your Databricks account, workspace, and the signed-in user all meet the requirements for Partner Connect.
In the sidebar, click Partner Connect.
Click the Prophecy tile.
If the Prophecy tile has a check mark icon inside of it, this means one of your administrators has already used Partner Connect to create an execution environment (Prophecy calls this a fabric) for your workspace. Contact that admin, who can add you to the Prophecy account that they created by using Partner Connect. After they add you, click the Prophecy tile.
If the Connect to partner dialog displays a Next button, click it.
Partner Connect creates a Databricks service principal in your workspace. This service principal is named PROPHECY_USER.
For Email, enter the email address that you want Prophecy to use to create a 14-day trial Prophecy account for you, or to sign in to your existing Prophecy account.
Click the button with the label Connect to Prophecy or Visit Prophecy.
If an error displays stating that someone from your organization has already created an account with Prophecy, do one of the following:
Enter an email address that is not associated with your organization, and then click Connect to Prophecy or Visit Prophecy again.
Contact one of your organization’s administrators and have them add you to your organization’s Prophecy account. After they add you, click Connect to Prophecy or Visit Prophecy again.
If you clicked the button with the label Connect to Prophecy, Partner Connect creates a Databricks personal access token and associates it with the PROPHECY_USER service principal.
On the separate Prophecy website tab that appears, complete the on-screen instructions to have Prophecy finish creating a 14-day trial Prophecy account for you, or sign in to your existing Prophecy account.
In your Prophecy account, on the navigation bar, click Metadata.
Click Fabrics. The fabric for your workspace should be displayed.
Do one of the following:
Use the following instructions to connect Prophecy to your workspace.
To connect faster to Prophecy, use Partner Connect.
To complete this series of steps, you will need a Databricks personal access token.
Sign in to your Prophecy account, or create a new Prophecy account, at https://app.prophecy.io.
On the navigation bar, click Metadata.
Click the Fabrics tab.
If you sign in to your organization’s Prophecy account, there may already be a list of existing fabric entries. These entries might contain connection details for workspaces that are separate from yours. If you still want to reuse one of these fabrics, and you trust the workspace and have access to it, then skip ahead to Next steps.
Click the plus (+) icon.
Click the Fabric tab.
Enter a Name, select a Team, and enter an optional Description. (Do not change the Execution Url value.)
Click Create Fabric.
In the list of fabrics, click the name of the fabric that you just added.
Follow the instructions for How to Configure a Fabric for Databricks on the Prophecy website.