Monitoring and observability options for Delta Live Tables pipelines

This article introduces the features included in Delta Live Tables that support monitoring, reporting, and auditing your pipelines, and generating alerts and notifications for pipeline events.

Delta Live Tables includes several features to support monitoring and observability of pipelines. These features support tasks such as:

• Observing the progress and status of pipeline updates.

• Alerts for pipeline events such as success or failure of pipeline updates.

• Extracting detailed information on pipeline updates such as data lineage, data quality metrics, and resource usage.

• Defining custom actions to take when specific events occur.

The articles in this section are detailed references to help you effectively use these features to monitor your Delta Live Tables pipelines.

Monitor pipelines with the Delta Live Tables UI and logs

You can use the pipeline details UI to view the status and progress of pipeline updates, a history of updates, and details about the datasets in a pipeline. For more advanced reporting and auditing tasks, you can use the Delta Live Tables event log.

See Monitor Delta Live Tables pipelines.

Implement custom monitoring of Delta Live Tables pipelines with event hooks (Public Preview)

You can use event hooks to implement custom monitoring and alerting for pipelines. For example, you can use event hooks to send emails or write to a log when specific events occur, or integrate with third-party solutions. Event hooks are custom Python callback functions that run when events are persisted to a Delta Live Tables pipeline’s event log.

See Define custom monitoring of Delta Live Tables pipelines with event hooks.