Monitor pipelines

Monitor Lakeflow pipelines using the built-in pipeline UI, the event log, query history, and custom event hooks. These features track update progress, data quality, lineage, and streaming metrics.

Pipeline monitoring works in three layers, from a quick glance to deep programmatic queries:

• The Jobs & Pipelines list is the fastest check. It shows the result of the last five runs as status indicators next to each pipeline, so you can tell at a glance whether recent runs succeeded, failed, or are still running.
• The pipeline monitoring UI shows the pipeline graph color-coded by status, per-table row counts and data quality metrics, update history, and streaming backlog metrics for the selected update. See Monitor pipelines in the UI.
• The event log is the ground truth underneath both. It's a structured Delta table that you query for anything programmatic or historical, such as update outcomes, data quality trends, and resource usage. See Pipeline event log.
• The pipeline_events system table (Beta) aggregates events across all pipelines across all workspaces in a region, so you can track fleet-wide health and trends with SQL. See Monitor and query events.

To alert on pipeline events, configure notifications. Set notifications directly on a pipeline when it runs on its own schedule, or as job notifications when the pipeline runs inside a Lakeflow Job. See Add email notifications for pipeline events.

Topic	Description
Monitor using the UI	Observe the progress and status of pipeline updates, and alert on the success or failure. View metrics for streaming sources, like Apache Kafka and Auto Loader.
Monitor and query events (Beta)	Track the health of all pipelines across all workspaces in a region with the pipeline events system table.
Event log	Extract detailed information on pipeline updates such as data lineage, data quality metrics, and resource usage using the pipeline event log. Additionally, see the schema for the event log.
Query history	Inspect and diagnose query performance by looking at the query history.
Custom monitoring	Define custom actions to take when specific events occur using event hooks.

Topic	Description
Monitor using the UI	Observe the progress and status of pipeline updates, and alert on the success or failure. View metrics for streaming sources, like Apache Kafka and Auto Loader.
Monitor and query events (Beta)	Track the health of all pipelines across all workspaces in a region with the pipeline events system table.
Event log	Extract detailed information on pipeline updates such as data lineage, data quality metrics, and resource usage using the pipeline event log. Additionally, see the schema for the event log.
Query history	Inspect and diagnose query performance by looking at the query history.
Custom monitoring	Define custom actions to take when specific events occur using event hooks.

Additionally, there are troubleshooting topics for specific scenarios.

Topic	Description
Recover a pipeline from streaming checkpoint failure	Recover a pipeline that has an invalid or corrupted streaming checkpoint.
Fix high initialization times in pipelines	Fix high initialization times for a pipeline by splitting and load balancing flows across pipelines.

Topic	Description
Recover a pipeline from streaming checkpoint failure	Recover a pipeline that has an invalid or corrupted streaming checkpoint.
Fix high initialization times in pipelines	Fix high initialization times for a pipeline by splitting and load balancing flows across pipelines.