Prophecy helps teams be successful and productive on Apache Spark and Apache Airflow with low-code development, scheduling, and metadata.
You can integrate your Databricks clusters with Prophecy.
For a general overview and demonstration of Prophecy, watch the following YouTube video (26 minutes).
Prophecy does not integrate with Databricks SQL warehouses (formerly Databricks SQL endpoints).
For an overview of the Partner Connect procedure, watch this YouTube video (3 minutes).
In your Prophecy account, on the navigation bar, click Metadata.
Click Fabrics. The fabric for your workspace should be displayed.
Do one of the following:
If the fabric is displayed, skip ahead to Next steps.
If the fabric is not displayed, you can troubleshoot by connecting manually.
Use the steps in this section to connect Prophecy to your workspace.
To connect faster to Prophecy, use Partner Connect.
To complete this series of steps, you need a Databricks personal access token.
As a security best practice, when authenticating with automated tools, systems, scripts, and apps, Databricks recommends you use access tokens belonging to service principals instead of workspace users. For more information, see Service principals for Databricks automation.
To connect to Prophecy manually, do the following:
Sign in to your Prophecy account, or create a new Prophecy account, at https://app.prophecy.io.
On the navigation bar, click Metadata.
Click the Fabrics tab.
If you sign in to your organization’s Prophecy account, there may already be a list of existing fabric entries. These entries might contain connection details for workspaces that are separate from yours. If you still want to reuse one of these fabrics, and you trust the workspace and have access to it, then skip ahead to Next steps.
Click the plus (+) icon.
Click the Fabric tab.
Enter a Name, select a Team, and enter an optional Description. (Do not change the Execution Url value.)
Click Create Fabric.
In the list of fabrics, click the name of the fabric that you just added.
Follow the instructions for How to Configure a Fabric for Databricks on the Prophecy website.
See Creating a Pipeline in the Prophecy documentation.