Pipeline events system table reference
This system table is in Beta.
This article is a reference for the pipeline_events system table, which records Lakeflow pipelines event log entries for pipelines in your account. Each row is an immutable event from the pipeline event log, capturing lifecycle transitions, flow progress, data quality metrics, errors, cluster resources, and other operational data across all pipelines and workspaces within a region.
Requirements
- To access this system table, users must either:
- Be both a metastore admin and an account admin, or
- Have
USEandSELECTpermissions on the system schemas. See Grant access to system tables.
Available pipeline event tables
The pipeline events system table lives in the lakeflow_pipeline_events_preview schema during Beta, and moves to the lakeflow schema at general availability:
Table | Description | Supports streaming | Free retention period | Includes global or regional data |
|---|---|---|---|---|
pipeline_events (Beta) | Records pipeline event log entries emitted by pipeline runs | Yes | 13 months | Regional |
The schema is lakeflow_pipeline_events_preview during Beta. At general availability, the table moves to the lakeflow schema (the final table path will be system.lakeflow.pipeline_events). Queries written against the Beta schema must be updated when the table moves.
Detailed schema reference
Pipeline events table schema
The pipeline events table is append-only. Each row records a single event emitted by a pipeline update at the time it was emitted, and rows are never modified or deleted in place.
Which fields are populated on a row depends on the event type. error, update_id, and many origin.* sub-fields are set only on events where they apply, and the structure of the details field also varies by event_type.
Use this table to query historical pipeline activity, to build alerts on pipeline failures, and to correlate pipeline behavior with other Lakeflow system tables.
Table path: system.lakeflow_pipeline_events_preview.pipeline_events
Primary key: (account_id, pipeline_event_id)
Column name | Data type | Description | Notes |
|---|---|---|---|
| string | The ID of the account this pipeline event belongs to | |
| string | The ID of the workspace this pipeline event belongs to | |
| string | The ID of the pipeline that emitted the event | |
| string | The ID of the pipeline update that emitted the event | |
| string | Globally unique identifier for the event | |
| string | The type of event (for example, | See Event type values for the full set of values. |
| struct | Contextual metadata about the origin of the event like cloud provider, region, pipeline type, table or flow names, and other identifiers | See Origin struct fields. |
| string | Human-readable description of the event | May be empty for some events. |
| string | Severity level of the event | One of |
| string | Stability of the event schema | One of |
| struct | Error details. Populated only for events that carry error information | See Error struct fields. |
| variant | Event-specific payload. The fields it contains depend on the | See Details field. |
| timestamp | The time the event was emitted by the pipeline | Timezone recorded as |
Origin struct fields
Sub-field | Data type | Description |
|---|---|---|
| string | Cloud provider (for example, |
| string | Cloud provider region |
| bigint | Workspace organization ID |
| string | The type of pipeline |
| string | The user-supplied name of the pipeline |
| string | The compute cluster ID backing the pipeline update |
| string | The ID of the maintenance update, if the event is from a maintenance run |
| string | The name of the dataset (table or view) the event refers to |
| string | The name of the sink the event refers to |
| string | The Unity Catalog catalog name |
| string | The Unity Catalog schema name |
| string | The ID of the flow the event refers to |
| string | The name of the flow the event refers to |
| bigint | The micro-batch ID for streaming flows. |
| string | The request ID that initiated the action |
| string | The materialization name |
| string | The operation ID |
| string | The name of the data source |
| string | The Unity Catalog table ID |
| string | The type of ingestion source (for example, |
| string | The connection name for the ingestion source |
| string | The source catalog name in the upstream system |
| string | The source schema name in the upstream system |
| string | The source table name in the upstream system |
| string | The source table version, where applicable |
Error struct fields
Sub-field | Data type | Description |
|---|---|---|
| boolean | Whether the error caused the update to terminate |
| array<struct> | Chain of exceptions associated with the error (root cause last) |
| string | SQLSTATE code, if available |
| string | Databricks error class, if available |
Details field
The details column is a VARIANT, and the fields it contains depend on the event_type. For the fields available under each event type, see Pipeline event log schema. Use the variant_get function or dot syntax to read nested values. See the example queries below for typical access patterns.
The event_type key wraps the payload. For example, a flow_progress event's metrics are at $.flow_progress.metrics, not $.metrics. Include the event-type key in every path.
-- Using variant_get (lets you cast to a specific type)
SELECT
pipeline_id,
event_time,
variant_get(details, '$.flow_progress.status', 'STRING') AS flow_status,
variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT') AS rows_written,
variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT') AS backlog_bytes
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
-- Using dot syntax (returns VARIANT, cast when needed)
SELECT
pipeline_id,
event_time,
details:flow_progress.status::STRING AS flow_status,
details:flow_progress.metrics.num_output_rows::BIGINT AS rows_written
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
Example queries
-- Flow throughput for a specific pipeline
SELECT
origin.flow_name,
date_trunc('HOUR', event_time) AS hour,
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
origin.flow_name,
date_trunc('HOUR', event_time)
ORDER BY
hour DESC,
rows_written DESC
-- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception.
-- The exception chain is ordered with the root cause last, so read element -1 for the root cause.
-- On many errors only the first element carries error_class and sql_state.
SELECT
workspace_id,
pipeline_id,
event_time,
event_type,
message,
error.exceptions[0].error_class AS exception_error_class,
error.exceptions[0].sql_state AS exception_sql_state
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
level = 'ERROR'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
event_time DESC
-- Data quality: failed expectations by dataset, per update, in the last 1 day
SELECT
pipeline_id,
update_id,
origin.dataset_name,
expectation.name AS expectation_name,
SUM(expectation.failed_records) AS failed_records
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY<STRUCT<name:STRING,dataset:STRING,passed_records:BIGINT,failed_records:BIGINT>>')) AS expectation
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY
pipeline_id,
update_id,
origin.dataset_name,
expectation.name
HAVING
SUM(expectation.failed_records) > 0
ORDER BY
failed_records DESC
Common join patterns
Join with the pipelines table to filter by pipeline name
The pipelines table is a slowly changing dimension (SCD2). Take the latest version of each pipeline before joining.
WITH latest_pipelines AS (
SELECT *
FROM system.lakeflow.pipelines
QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY change_time DESC) = 1
)
SELECT
p.name AS pipeline_name,
e.event_time,
e.event_type,
e.level,
e.message
FROM
system.lakeflow_pipeline_events_preview.pipeline_events e
JOIN
latest_pipelines p
ON e.workspace_id = p.workspace_id
AND e.pipeline_id = p.pipeline_id
WHERE
e.level = 'ERROR'
AND e.event_time >= current_timestamp() - INTERVAL 24 HOURS
ORDER BY
e.event_time DESC
Join with pipeline_update_timeline on update_id
SELECT
u.period_start_time AS update_start,
u.period_end_time AS update_end,
e.event_time,
e.event_type,
e.level,
e.message
FROM
system.lakeflow.pipeline_update_timeline u
JOIN
system.lakeflow_pipeline_events_preview.pipeline_events e
ON e.update_id = u.update_id
WHERE
u.pipeline_id = '<your-pipeline-id>'
AND u.period_start_time >= current_timestamp() - INTERVAL 7 DAYS
ORDER BY
u.period_start_time DESC,
e.event_time ASC
Setting up alerts
You can build alerts on pipeline_events using Databricks SQL alerts. Write a SQL query against pipeline_events (optionally joined with other Lakeflow system tables), schedule it on a SQL warehouse, and configure a notification destination (email, Slack, webhook, PagerDuty).
A few useful starting points:
Alert when no events arrived for a pipeline in the last N minutes
Use this to detect stuck or silently failing pipelines.
-- Returns one row per pipeline that has not emitted any event in the last 30 minutes.
-- The alert can trigger when this query returns any rows.
SELECT
workspace_id,
pipeline_id,
MAX(event_time) AS last_event_time
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_time >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY
workspace_id,
pipeline_id
HAVING
MAX(event_time) < current_timestamp() - INTERVAL 30 MINUTES
Alert when the backlog for a specific flow is too high
Backlog is reported on flow_progress events as backlog_bytes, and for file sources also as backlog_files. Trigger when the most recent reading crosses a threshold (for example, 100 MB of unprocessed work). Not every source reports every metric, so filter on the one your source populates.
-- Returns the most recent backlog reading per flow for a given pipeline.
-- The alert can trigger when backlog_bytes exceeds the threshold for any flow.
WITH latest_flow_progress AS (
SELECT
workspace_id,
pipeline_id,
origin.flow_name,
event_time,
CASE
WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
ELSE variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT')
END AS backlog_bytes
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND (
variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT') IS NOT NULL
OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
)
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_flow_progress
WHERE backlog_bytes > 100000000
To see which source is behind, $.flow_progress.metrics.source_metrics is an array of per-source readings, each with source_name alongside that source's backlog_bytes, backlog_records or backlog_files.
Alert on data-quality drops in a pipeline
Each flow_progress event reports the number of rows dropped by EXPECT … DROP expectations. Sum these per dataset over an update window and alert when the total exceeds a threshold.
-- Returns datasets where more than 100 rows were dropped by expectations, per update, in the last hour.
SELECT
pipeline_id,
update_id,
origin.dataset_name,
SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) AS dropped_records
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
GROUP BY
pipeline_id,
update_id,
origin.dataset_name
HAVING
SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) > 100
Alert when a flow processes too few rows
flow_progress events report metrics.num_output_rows as a per-micro-batch count, so summing the events in a window gives the rows written over that window. Create an alert for when throughput drops below an expected floor. For example, a flow that normally writes thousands of rows per hour but produces near zero can indicate a misconfigured source.
This query only reports flows that emitted a flow_progress event with a row count in the window. A fully stalled flow emits no events, so pair this alert with the missing-events alert above.
-- Returns flows that wrote fewer than 100 rows in the last hour.
-- The alert can trigger when this query returns any rows.
SELECT
pipeline_id,
origin.flow_name,
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written_last_hour
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT') IS NOT NULL
GROUP BY
pipeline_id,
origin.flow_name
HAVING
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) < 100
Alert when new-data latency is too high
For streaming flows, flow_progress events report latency in streaming_metrics. stream_latency_ms is the time from when data landed upstream to when the micro-batch committed to the Delta table. You can set a trigger for when the most recent reading crosses a threshold (for example, 5 minutes).
Only streaming flows with a tagged event time report stream_latency_ms, and only when SDP time-metrics are enabled. Other flows return NULL on every event, and this alert never fires for them.
-- Returns the most recent new-data latency per flow for a given pipeline.
-- The alert can trigger when stream_latency_ms > 300000 (5 minutes) for any flow.
WITH latest_latency AS (
SELECT
workspace_id,
pipeline_id,
origin.flow_name,
event_time,
CASE
WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
ELSE variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT')
END AS stream_latency_ms
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND (
variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT') IS NOT NULL
OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
)
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_latency
WHERE stream_latency_ms > 300000
Tips for production alerts
- Filter by
pipeline_id(andworkspace_idif you maintain alerts per workspace) so each alert targets a specific scope rather than the whole account. - Choose an evaluation cadence that matches the alert's sensitivity. Use a short interval (for example, every 5 minutes) for fast-fail signals and a longer interval (for example, hourly) for backlog and data-quality trends. A "trigger when query returns more than 0 rows" condition works for most cases.
Reference values
Level values
Value | Description |
|---|---|
| Normal pipeline activity (flow progress, update lifecycle transitions, configuration changes). |
| Non-fatal issues the pipeline recovered from, or that may require attention. |
| Failures that prevented the pipeline from making progress on a flow or update. |
| Quantitative measurements emitted during execution (row counts, throughput, latency). |
Maturity level values
Value | Description |
|---|---|
| The event schema is stable. Breaking changes are not expected. Safe to build production queries and alerts on. |
| The event schema may change in future releases. Use with care. |
| The event type or schema is deprecated and will be removed in future releases. Migrate off it. |
Event type values
The event_type field is an enumeration. The full set of values:
Value | Description |
|---|---|
| A new pipeline update was requested. |
| A pipeline update transitioned through a lifecycle state. |
| A flow (dataset) within an update transitioned through a state. |
| Static metadata about a flow. |
| Static metadata about a dataset. |
| Static metadata about an output sink. |
| A deprecated feature was used by the pipeline. |
| Cluster autoscaling decision. |
| An operation that is not supported in the current configuration. |
| Task slot and autoscale metrics for the backing compute. |
| Planning-phase information for the update. |
| Garbage collection pressure on driver or executors. |
| The update terminated abnormally. |
| Disk space pressure on the cluster. |
| Lifecycle progress of a pipeline hook. |
| A dataset lifecycle event. |
| A background operation transitioned through a state. |
| The pipeline made an outbound API call. |
| Progress for a generic operation. |
| Progress for a streaming query backing a flow. |
| Summary of a pipeline rewind operation. |
| An advisory message from the engine. |
| Detailed runtime configuration. |
| Resource information (cluster, instance type, etc.). |
| File notification setup status (for cloud-files sources). |
| Behavior-change notification under Spark Connect. |
| A user-initiated action against the pipeline. |
| Context about the user code associated with the event. |