Develop and debug Delta Live Tables pipelines in notebooks
Preview
The notebook experience for Delta Live Tables development is in Public Preview.
This article describes features in Databricks notebooks that assist in the development and debugging of Delta Live Tables code.
Overview of features
When you work on a Python or SQL notebook that is the source code for an existing Delta Live Tables pipeline, you can connect the notebook directly to the pipeline. When the notebook is connected to the pipeline, the following features are available:
Start and validate the pipeline from the notebook.
View the pipeline’s dataflow graph and event log for the latest update in the notebook.
View pipeline diagnostics in the notebook editor.
View the status of the pipeline’s cluster in the notebook.
Access the Delta Live Tables UI from the notebook.
Prerequisites
You must have an existing Delta Live Tables pipeline with a Python or SQL notebook as source code.
You must either be the owner of the pipeline or have the
CAN_MANAGE
privilege.
Limitations
The features covered in this article are only available in Databricks notebooks. Workspace files are not supported.
The web terminal is not available when attached to a pipeline. As a result, it is not visible as a tab in the bottom panel.
Connect a notebook to a Delta Live Tables pipeline
Inside the notebook, click on the drop-down menu used to select compute. The drop-down menu shows all your Delta Live Tables pipelines with this notebook as source code. To connect the notebook to a pipeline, select it from the list.
View the pipeline’s cluster status
To easily understand the state of your pipeline’s cluster, its status is shown in the compute drop-down menu with a green color to indicate that the cluster is running.
Validate pipeline code
You can validate the pipeline to check for syntax errors in your source code without processing any data.
To validate a pipeline, do one of the following:
In the top-right corner of the notebook, click Validate.
Press
Shift+Enter
in any notebook cell.In a cell’s dropdown menu, click Validate Pipeline.
Note
If you attempt to validate your pipeline while an existing update is already running, a dialog box displays asking if you want to terminate the existing update.
Start the pipeline
A pipeline update does the following: starts a cluster, discovers and validates all the tables and views defined, and creates or updates tables and views with the most recent data available.
To start an update of your pipeline, click the Start button in the top-right corner of the notebook.
If you click Yes, the existing update stops, and a validate update automatically starts.
View the status of an update
The top panel in the notebook displays whether a pipeline update is:
Starting
Validating
Stopping
View errors and diagnostics
After a pipeline has been started or validated, any errors are shown inline with a red underline. Hover over an error to see more information.
View pipeline events
When attached to a pipeline, there is a Delta Live Tables event log tab at the bottom of the notebook.
View the pipeline Dataflow Graph
To view a pipeline’s dataflow graph, use the Delta Live Tables graph tab at the bottom of the notebook. Selecting a node in the graph displays its schema in the right panel.