Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Pipeline events system table reference

Beta

This system table is in Beta.

This article is a reference for the pipeline_events system table, which records Lakeflow pipelines event log entries for pipelines in your account. Each row is an immutable event from the pipeline event log, capturing lifecycle transitions, flow progress, data quality metrics, errors, cluster resources, and other operational data across all pipelines and workspaces within a region.

Requirements

  • To access this system table, users must either:

Available pipeline event tables

The pipeline events system table lives in the lakeflow_pipeline_events_preview schema during Beta, and moves to the lakeflow schema at general availability:

Table

Description

Supports streaming

Free retention period

Includes global or regional data

pipeline_events (Beta)

Records pipeline event log entries emitted by pipeline runs

Yes

13 months

Regional

Table

Description

Supports streaming

Free retention period

Includes global or regional data

pipeline_events (Beta)

Records pipeline event log entries emitted by pipeline runs

Yes

13 months

Regional

note

The schema is lakeflow_pipeline_events_preview during Beta. At general availability, the table moves to the lakeflow schema (the final table path will be system.lakeflow.pipeline_events). Queries written against the Beta schema must be updated when the table moves.

Detailed schema reference

Pipeline events table schema

The pipeline events table is append-only. Each row records a single event emitted by a pipeline update at the time it was emitted, and rows are never modified or deleted in place.

Which fields are populated on a row depends on the event type. error, update_id, and many origin.* sub-fields are set only on events where they apply, and the structure of the details field also varies by event_type.

Use this table to query historical pipeline activity, to build alerts on pipeline failures, and to correlate pipeline behavior with other Lakeflow system tables.

Table path: system.lakeflow_pipeline_events_preview.pipeline_events

Primary key: (account_id, pipeline_event_id)

Column name

Data type

Description

Notes

account_id

string

The ID of the account this pipeline event belongs to

workspace_id

string

The ID of the workspace this pipeline event belongs to

pipeline_id

string

The ID of the pipeline that emitted the event

update_id

string

The ID of the pipeline update that emitted the event

pipeline_event_id

string

Globally unique identifier for the event

event_type

string

The type of event (for example, flow_progress, update_progress, create_update)

See Event type values for the full set of values.

origin

struct

Contextual metadata about the origin of the event like cloud provider, region, pipeline type, table or flow names, and other identifiers

See Origin struct fields.

message

string

Human-readable description of the event

May be empty for some events.

level

string

Severity level of the event

One of INFO, WARN, ERROR, METRICS. See Level values.

maturity_level

string

Stability of the event schema

One of STABLE, EVOLVING, DEPRECATED. See Maturity level values.

error

struct

Error details. Populated only for events that carry error information

See Error struct fields.

details

variant

Event-specific payload. The fields it contains depend on the event_type

See Details field.

event_time

timestamp

The time the event was emitted by the pipeline

Timezone recorded as +00:00 (UTC).

Column name

Data type

Description

Notes

account_id

string

The ID of the account this pipeline event belongs to

workspace_id

string

The ID of the workspace this pipeline event belongs to

pipeline_id

string

The ID of the pipeline that emitted the event

update_id

string

The ID of the pipeline update that emitted the event

pipeline_event_id

string

Globally unique identifier for the event

event_type

string

The type of event (for example, flow_progress, update_progress, create_update)

See Event type values for the full set of values.

origin

struct

Contextual metadata about the origin of the event like cloud provider, region, pipeline type, table or flow names, and other identifiers

See Origin struct fields.

message

string

Human-readable description of the event

May be empty for some events.

level

string

Severity level of the event

One of INFO, WARN, ERROR, METRICS. See Level values.

maturity_level

string

Stability of the event schema

One of STABLE, EVOLVING, DEPRECATED. See Maturity level values.

error

struct

Error details. Populated only for events that carry error information

See Error struct fields.

details

variant

Event-specific payload. The fields it contains depend on the event_type

See Details field.

event_time

timestamp

The time the event was emitted by the pipeline

Timezone recorded as +00:00 (UTC).

Origin struct fields

Sub-field

Data type

Description

cloud

string

Cloud provider (for example, AWS, AZURE, GCP)

region

string

Cloud provider region

org_id

bigint

Workspace organization ID

pipeline_type

string

The type of pipeline

pipeline_name

string

The user-supplied name of the pipeline

cluster_id

string

The compute cluster ID backing the pipeline update

maintenance_id

string

The ID of the maintenance update, if the event is from a maintenance run

dataset_name

string

The name of the dataset (table or view) the event refers to

sink_name

string

The name of the sink the event refers to

catalog_name

string

The Unity Catalog catalog name

schema_name

string

The Unity Catalog schema name

flow_id

string

The ID of the flow the event refers to

flow_name

string

The name of the flow the event refers to

batch_id

bigint

The micro-batch ID for streaming flows. bigint for compatibility

request_id

string

The request ID that initiated the action

materialization_name

string

The materialization name

operation_id

string

The operation ID

source_name

string

The name of the data source

uc_table_id

string

The Unity Catalog table ID

ingestion_source_type

string

The type of ingestion source (for example, SQL_SERVER, SALESFORCE)

ingestion_source_connection_name

string

The connection name for the ingestion source

ingestion_source_catalog_name

string

The source catalog name in the upstream system

ingestion_source_schema_name

string

The source schema name in the upstream system

ingestion_source_table_name

string

The source table name in the upstream system

ingestion_source_table_version

string

The source table version, where applicable

Sub-field

Data type

Description

cloud

string

Cloud provider (for example, AWS, AZURE, GCP)

region

string

Cloud provider region

org_id

bigint

Workspace organization ID

pipeline_type

string

The type of pipeline

pipeline_name

string

The user-supplied name of the pipeline

cluster_id

string

The compute cluster ID backing the pipeline update

maintenance_id

string

The ID of the maintenance update, if the event is from a maintenance run

dataset_name

string

The name of the dataset (table or view) the event refers to

sink_name

string

The name of the sink the event refers to

catalog_name

string

The Unity Catalog catalog name

schema_name

string

The Unity Catalog schema name

flow_id

string

The ID of the flow the event refers to

flow_name

string

The name of the flow the event refers to

batch_id

bigint

The micro-batch ID for streaming flows. bigint for compatibility

request_id

string

The request ID that initiated the action

materialization_name

string

The materialization name

operation_id

string

The operation ID

source_name

string

The name of the data source

uc_table_id

string

The Unity Catalog table ID

ingestion_source_type

string

The type of ingestion source (for example, SQL_SERVER, SALESFORCE)

ingestion_source_connection_name

string

The connection name for the ingestion source

ingestion_source_catalog_name

string

The source catalog name in the upstream system

ingestion_source_schema_name

string

The source schema name in the upstream system

ingestion_source_table_name

string

The source table name in the upstream system

ingestion_source_table_version

string

The source table version, where applicable

Error struct fields

Sub-field

Data type

Description

fatal

boolean

Whether the error caused the update to terminate

exceptions

array<struct>

Chain of exceptions associated with the error (root cause last)

exceptions[].sql_state

string

SQLSTATE code, if available

exceptions[].error_class

string

Databricks error class, if available

Sub-field

Data type

Description

fatal

boolean

Whether the error caused the update to terminate

exceptions

array<struct>

Chain of exceptions associated with the error (root cause last)

exceptions[].sql_state

string

SQLSTATE code, if available

exceptions[].error_class

string

Databricks error class, if available

Details field

The details column is a VARIANT, and the fields it contains depend on the event_type. For the fields available under each event type, see Pipeline event log schema. Use the variant_get function or dot syntax to read nested values. See the example queries below for typical access patterns.

The event_type key wraps the payload. For example, a flow_progress event's metrics are at $.flow_progress.metrics, not $.metrics. Include the event-type key in every path.

SQL
-- Using variant_get (lets you cast to a specific type)
SELECT
pipeline_id,
event_time,
variant_get(details, '$.flow_progress.status', 'STRING') AS flow_status,
variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT') AS rows_written,
variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT') AS backlog_bytes
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
SQL
-- Using dot syntax (returns VARIANT, cast when needed)
SELECT
pipeline_id,
event_time,
details:flow_progress.status::STRING AS flow_status,
details:flow_progress.metrics.num_output_rows::BIGINT AS rows_written
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR

Example queries

SQL
-- Flow throughput for a specific pipeline
SELECT
origin.flow_name,
date_trunc('HOUR', event_time) AS hour,
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
origin.flow_name,
date_trunc('HOUR', event_time)
ORDER BY
hour DESC,
rows_written DESC
SQL
-- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception.
-- The exception chain is ordered with the root cause last, so read element -1 for the root cause.
-- On many errors only the first element carries error_class and sql_state.
SELECT
workspace_id,
pipeline_id,
event_time,
event_type,
message,
error.exceptions[0].error_class AS exception_error_class,
error.exceptions[0].sql_state AS exception_sql_state
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
level = 'ERROR'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
event_time DESC
SQL
-- Data quality: failed expectations by dataset, per update, in the last 1 day
SELECT
pipeline_id,
update_id,
origin.dataset_name,
expectation.name AS expectation_name,
SUM(expectation.failed_records) AS failed_records
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY<STRUCT<name:STRING,dataset:STRING,passed_records:BIGINT,failed_records:BIGINT>>')) AS expectation
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY
pipeline_id,
update_id,
origin.dataset_name,
expectation.name
HAVING
SUM(expectation.failed_records) > 0
ORDER BY
failed_records DESC

Common join patterns

Join with the pipelines table to filter by pipeline name

The pipelines table is a slowly changing dimension (SCD2). Take the latest version of each pipeline before joining.

SQL
WITH latest_pipelines AS (
SELECT *
FROM system.lakeflow.pipelines
QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY change_time DESC) = 1
)
SELECT
p.name AS pipeline_name,
e.event_time,
e.event_type,
e.level,
e.message
FROM
system.lakeflow_pipeline_events_preview.pipeline_events e
JOIN
latest_pipelines p
ON e.workspace_id = p.workspace_id
AND e.pipeline_id = p.pipeline_id
WHERE
e.level = 'ERROR'
AND e.event_time >= current_timestamp() - INTERVAL 24 HOURS
ORDER BY
e.event_time DESC

Join with pipeline_update_timeline on update_id

SQL
SELECT
u.period_start_time AS update_start,
u.period_end_time AS update_end,
e.event_time,
e.event_type,
e.level,
e.message
FROM
system.lakeflow.pipeline_update_timeline u
JOIN
system.lakeflow_pipeline_events_preview.pipeline_events e
ON e.update_id = u.update_id
WHERE
u.pipeline_id = '<your-pipeline-id>'
AND u.period_start_time >= current_timestamp() - INTERVAL 7 DAYS
ORDER BY
u.period_start_time DESC,
e.event_time ASC

Setting up alerts

You can build alerts on pipeline_events using Databricks SQL alerts. Write a SQL query against pipeline_events (optionally joined with other Lakeflow system tables), schedule it on a SQL warehouse, and configure a notification destination (email, Slack, webhook, PagerDuty).

A few useful starting points:

Alert when no events arrived for a pipeline in the last N minutes

Use this to detect stuck or silently failing pipelines.

SQL
-- Returns one row per pipeline that has not emitted any event in the last 30 minutes.
-- The alert can trigger when this query returns any rows.
SELECT
workspace_id,
pipeline_id,
MAX(event_time) AS last_event_time
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_time >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY
workspace_id,
pipeline_id
HAVING
MAX(event_time) < current_timestamp() - INTERVAL 30 MINUTES

Alert when the backlog for a specific flow is too high

Backlog is reported on flow_progress events as backlog_bytes, and for file sources also as backlog_files. Trigger when the most recent reading crosses a threshold (for example, 100 MB of unprocessed work). Not every source reports every metric, so filter on the one your source populates.

SQL
-- Returns the most recent backlog reading per flow for a given pipeline.
-- The alert can trigger when backlog_bytes exceeds the threshold for any flow.
WITH latest_flow_progress AS (
SELECT
workspace_id,
pipeline_id,
origin.flow_name,
event_time,
CASE
WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
ELSE variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT')
END AS backlog_bytes
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND (
variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT') IS NOT NULL
OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
)
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_flow_progress
WHERE backlog_bytes > 100000000

To see which source is behind, $.flow_progress.metrics.source_metrics is an array of per-source readings, each with source_name alongside that source's backlog_bytes, backlog_records or backlog_files.

Alert on data-quality drops in a pipeline

Each flow_progress event reports the number of rows dropped by EXPECT … DROP expectations. Sum these per dataset over an update window and alert when the total exceeds a threshold.

SQL
-- Returns datasets where more than 100 rows were dropped by expectations, per update, in the last hour.
SELECT
pipeline_id,
update_id,
origin.dataset_name,
SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) AS dropped_records
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
GROUP BY
pipeline_id,
update_id,
origin.dataset_name
HAVING
SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) > 100

Alert when a flow processes too few rows

flow_progress events report metrics.num_output_rows as a per-micro-batch count, so summing the events in a window gives the rows written over that window. Create an alert for when throughput drops below an expected floor. For example, a flow that normally writes thousands of rows per hour but produces near zero can indicate a misconfigured source.

This query only reports flows that emitted a flow_progress event with a row count in the window. A fully stalled flow emits no events, so pair this alert with the missing-events alert above.

SQL
-- Returns flows that wrote fewer than 100 rows in the last hour.
-- The alert can trigger when this query returns any rows.
SELECT
pipeline_id,
origin.flow_name,
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written_last_hour
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT') IS NOT NULL
GROUP BY
pipeline_id,
origin.flow_name
HAVING
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) < 100

Alert when new-data latency is too high

For streaming flows, flow_progress events report latency in streaming_metrics. stream_latency_ms is the time from when data landed upstream to when the micro-batch committed to the Delta table. You can set a trigger for when the most recent reading crosses a threshold (for example, 5 minutes).

Only streaming flows with a tagged event time report stream_latency_ms, and only when SDP time-metrics are enabled. Other flows return NULL on every event, and this alert never fires for them.

SQL
-- Returns the most recent new-data latency per flow for a given pipeline.
-- The alert can trigger when stream_latency_ms > 300000 (5 minutes) for any flow.
WITH latest_latency AS (
SELECT
workspace_id,
pipeline_id,
origin.flow_name,
event_time,
CASE
WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
ELSE variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT')
END AS stream_latency_ms
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 HOUR
AND (
variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT') IS NOT NULL
OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
)
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_latency
WHERE stream_latency_ms > 300000

Tips for production alerts

  • Filter by pipeline_id (and workspace_id if you maintain alerts per workspace) so each alert targets a specific scope rather than the whole account.
  • Choose an evaluation cadence that matches the alert's sensitivity. Use a short interval (for example, every 5 minutes) for fast-fail signals and a longer interval (for example, hourly) for backlog and data-quality trends. A "trigger when query returns more than 0 rows" condition works for most cases.

Reference values

Level values

Value

Description

INFO

Normal pipeline activity (flow progress, update lifecycle transitions, configuration changes).

WARN

Non-fatal issues the pipeline recovered from, or that may require attention.

ERROR

Failures that prevented the pipeline from making progress on a flow or update.

METRICS

Quantitative measurements emitted during execution (row counts, throughput, latency).

Value

Description

INFO

Normal pipeline activity (flow progress, update lifecycle transitions, configuration changes).

WARN

Non-fatal issues the pipeline recovered from, or that may require attention.

ERROR

Failures that prevented the pipeline from making progress on a flow or update.

METRICS

Quantitative measurements emitted during execution (row counts, throughput, latency).

Maturity level values

Value

Description

STABLE

The event schema is stable. Breaking changes are not expected. Safe to build production queries and alerts on.

EVOLVING

The event schema may change in future releases. Use with care.

DEPRECATED

The event type or schema is deprecated and will be removed in future releases. Migrate off it.

Value

Description

STABLE

The event schema is stable. Breaking changes are not expected. Safe to build production queries and alerts on.

EVOLVING

The event schema may change in future releases. Use with care.

DEPRECATED

The event type or schema is deprecated and will be removed in future releases. Migrate off it.

Event type values

The event_type field is an enumeration. The full set of values:

Value

Description

create_update

A new pipeline update was requested.

update_progress

A pipeline update transitioned through a lifecycle state.

flow_progress

A flow (dataset) within an update transitioned through a state.

flow_definition

Static metadata about a flow.

dataset_definition

Static metadata about a dataset.

sink_definition

Static metadata about an output sink.

deprecation

A deprecated feature was used by the pipeline.

autoscale

Cluster autoscaling decision.

unsupported_operation

An operation that is not supported in the current configuration.

cluster_resources

Task slot and autoscale metrics for the backing compute.

planning_information

Planning-phase information for the update.

gc_pressure

Garbage collection pressure on driver or executors.

abnormal_termination

The update terminated abnormally.

disk_space

Disk space pressure on the cluster.

hook_progress

Lifecycle progress of a pipeline hook.

dataset_life_cycle

A dataset lifecycle event.

background_operation

A background operation transitioned through a state.

remote_api_usage

The pipeline made an outbound API call.

operation_progress

Progress for a generic operation.

stream_progress

Progress for a streaming query backing a flow.

rewind_summary

Summary of a pipeline rewind operation.

advisory

An advisory message from the engine.

runtime_details

Detailed runtime configuration.

resource_info

Resource information (cluster, instance type, etc.).

file_notification_set_up

File notification setup status (for cloud-files sources).

behavior_change_in_spark_connect

Behavior-change notification under Spark Connect.

user_action

A user-initiated action against the pipeline.

user_code_context

Context about the user code associated with the event.

Value

Description

create_update

A new pipeline update was requested.

update_progress

A pipeline update transitioned through a lifecycle state.

flow_progress

A flow (dataset) within an update transitioned through a state.

flow_definition

Static metadata about a flow.

dataset_definition

Static metadata about a dataset.

sink_definition

Static metadata about an output sink.

deprecation

A deprecated feature was used by the pipeline.

autoscale

Cluster autoscaling decision.

unsupported_operation

An operation that is not supported in the current configuration.

cluster_resources

Task slot and autoscale metrics for the backing compute.

planning_information

Planning-phase information for the update.

gc_pressure

Garbage collection pressure on driver or executors.

abnormal_termination

The update terminated abnormally.

disk_space

Disk space pressure on the cluster.

hook_progress

Lifecycle progress of a pipeline hook.

dataset_life_cycle

A dataset lifecycle event.

background_operation

A background operation transitioned through a state.

remote_api_usage

The pipeline made an outbound API call.

operation_progress

Progress for a generic operation.

stream_progress

Progress for a streaming query backing a flow.

rewind_summary

Summary of a pipeline rewind operation.

advisory

An advisory message from the engine.

runtime_details

Detailed runtime configuration.

resource_info

Resource information (cluster, instance type, etc.).

file_notification_set_up

File notification setup status (for cloud-files sources).

behavior_change_in_spark_connect

Behavior-change notification under Spark Connect.

user_action

A user-initiated action against the pipeline.

user_code_context

Context about the user code associated with the event.