Skip to main content

Monitor ingestion gateway progress with event logs

Applies to: Red X icon SaaS connectors Green check icon Database connectors

Learn how to use event logs to monitor the progress of ingestion gateways in real time. The gateway emits per-table metrics for both the snapshot and change data capture (CDC) phases, so you can track pipeline health, identify stalled tables, and build alerts and dashboards from structured event data.

Progress events allow you to:

  • Track how many rows and bytes have been ingested per table without waiting for pipeline completion.
  • Monitor snapshot progress and estimated time to completion for large initial loads.
  • Measure CDC latency from the source commit to the Unity Catalog volume for each table.
  • Confirm that the pipeline is actively running, even when no data changes occur.

How progress events work​

The gateway emits the following event types every five minutes for each table in your pipeline. The first events appear within five minutes of the pipeline starting.

  • flow_progress: Emitted for both snapshot and CDC flows. Reports rows upserted, rows deleted (CDC only), and output bytes since the previous event. For CDC flows, also reports latency metrics.
  • operation_progress: Emitted for snapshot flows only. Reports the snapshot completion percentage and the estimated time remaining until the snapshot finishes.

Events are emitted for every table, including tables with no changes since the previous event. A table that reports zero rows is idle. A table that reports no events hasn't been picked up by the pipeline.

Both event types use the METRICS log level. You can query them from the event log table using SQL.

Access progress events​

Progress events are stored in the event log table. To access them:

  1. Navigate to your gateway in the Databricks workspace.
  2. Click the Event log tab to view events in the UI.
  3. Query the event log table directly using SQL for detailed analysis.

Query the event log table​

To query flow_progress events for row and byte counters:

SQL
SELECT
timestamp,
CONCAT(origin.catalog_name, '.', origin.schema_name, '.', origin.dataset_name) AS table_name,
details:flow_progress:metrics:num_upserted_rows::bigint AS rows_upserted,
COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0) AS rows_deleted,
details:flow_progress:metrics:num_output_bytes::bigint AS output_bytes,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'snapshot'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'cdc'
ELSE 'unknown'
END AS ingestion_phase
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.pipeline_type = 'INGESTION_GATEWAY'
ORDER BY timestamp DESC

To query operation_progress events for snapshot progress percentage:

SQL
SELECT
timestamp,
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.pipeline_type = 'INGESTION_GATEWAY'
ORDER BY timestamp DESC

Replace <pipeline-id> with your gateway ID.

Understand the event structure​

The following examples show the JSON structure for each event type:

Snapshot flow progress event structure

JSON
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:14.175Z",
"level": "METRICS",
"event_type": "flow_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_snapshot_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Completed a streaming update of 'main.sales.customers_snapshot_flow'.",
"details": {
"flow_progress": {
"status": "RUNNING",
"metrics": {
"num_upserted_rows": 7512704,
"num_deleted_rows": null,
"num_output_bytes": 458752000
}
}
},
"maturity_level": "STABLE"
}

CDC flow progress event structure

JSON
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:57.426Z",
"level": "METRICS",
"event_type": "flow_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_cdc_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Completed a streaming update of 'main.sales.customers_cdc_flow'.",
"details": {
"flow_progress": {
"status": "RUNNING",
"metrics": {
"num_upserted_rows": 25,
"num_deleted_rows": 3,
"num_output_bytes": 18432
},
"streaming_metrics": {
"discovery_latency_ms": 12450,
"batch_processing_time_ms": 8100,
"event_time": {
"max": "2025-10-14T13:33:45.000Z"
}
}
}
},
"maturity_level": "STABLE"
}

Snapshot operation progress event structure

JSON
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:14.175Z",
"level": "METRICS",
"event_type": "operation_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_snapshot_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Snapshot in progress for 'main.sales.customers'.",
"details": {
"operation_progress": {
"type": "CDC_SNAPSHOT",
"status": "IN_PROGRESS",
"duration_ms": 3600000,
"progress_percent": 65.5,
"estimated_completion_ms": 1885000,
"cdc_snapshot": {
"target_table_name": "main.sales.customers",
"snapshot_timestamp": 1737542400000,
"snapshot_reason": "NEW_TABLE"
}
}
},
"maturity_level": "STABLE"
}

Event fields​

The following table describes the key fields in progress events:

Field

Type

Description

event_type

String

Either flow_progress (row and byte counters for snapshot and CDC) or operation_progress (snapshot progress percentage).

level

String

Always METRICS.

timestamp

String

ISO 8601 timestamp when the event was generated.

origin.pipeline_type

String

Always INGESTION_GATEWAY.

origin.pipeline_name

String

Name of the gateway.

origin.dataset_name

String

Name of the table being ingested.

origin.catalog_name

String

Unity Catalog catalog name.

origin.schema_name

String

Unity Catalog schema name.

origin.flow_name

String

Flow identifier that indicates the ingestion phase. Format: {catalog}.{schema}.{table}_snapshot_flow for initial load or {catalog}.{schema}.{table}_cdc_flow for incremental changes.

origin.ingestion_source_type

String

Source database type (for example, SQLSERVER, MYSQL, POSTGRESQL, ORACLE).

details:flow_progress:status

String

Current flow status, typically RUNNING.

details:flow_progress:metrics:num_upserted_rows

Integer

Number of rows inserted or updated since the last event. Delta metric. See Row and byte counters.

details:flow_progress:metrics:num_deleted_rows

Integer

Number of rows deleted since the last event. Delta metric. null for snapshot flows.

details:flow_progress:metrics:num_output_bytes

Integer

Number of compressed bytes uploaded to a volume since the last event. Delta metric. Not reported for query-based connector snapshots. See Query-based connector snapshots.

details:flow_progress:streaming_metrics:discovery_latency_ms

Integer

Time in milliseconds between the source change at event_time:max and when this event was emitted. CDC flows only. Can be null if the source does not expose change timestamps.

details:flow_progress:streaming_metrics:batch_processing_time_ms

Integer

Time in milliseconds the gateway spent reading and uploading the most recent batch of changes. Does not include source-side lag. CDC flows only. Can be null until the first batch completes.

details:flow_progress:streaming_metrics:event_time:max

String

ISO 8601 timestamp of the most recent source change the gateway has read for this table. CDC flows only.

details:operation_progress:type

String

Operation type. CDC_SNAPSHOT for snapshot operations.

details:operation_progress:status

String

Current operation status. IN_PROGRESS while the snapshot is running, COMPLETED when it finishes. Other values include STARTED, CANCELED, and FAILED. Gateways running a query-based connector snapshot also report SCHEDULED while a table is queued. See Query-based connector snapshots.

details:operation_progress:duration_ms

Integer

Total elapsed time of the operation in milliseconds.

details:operation_progress:progress_percent

Double

Snapshot completion percentage (0.0 - 100.0). Cumulative. See Snapshot progress and ETA.

details:operation_progress:estimated_completion_ms

Integer

Estimated time remaining in milliseconds until the snapshot finishes. See Snapshot progress and ETA.

details:operation_progress:cdc_snapshot:target_table_name

String

Fully qualified name of the table being snapshotted.

maturity_level

String

Always STABLE.

Field

Type

Description

event_type

String

Either flow_progress (row and byte counters for snapshot and CDC) or operation_progress (snapshot progress percentage).

level

String

Always METRICS.

timestamp

String

ISO 8601 timestamp when the event was generated.

origin.pipeline_type

String

Always INGESTION_GATEWAY.

origin.pipeline_name

String

Name of the gateway.

origin.dataset_name

String

Name of the table being ingested.

origin.catalog_name

String

Unity Catalog catalog name.

origin.schema_name

String

Unity Catalog schema name.

origin.flow_name

String

Flow identifier that indicates the ingestion phase. Format: {catalog}.{schema}.{table}_snapshot_flow for initial load or {catalog}.{schema}.{table}_cdc_flow for incremental changes.

origin.ingestion_source_type

String

Source database type (for example, SQLSERVER, MYSQL, POSTGRESQL, ORACLE).

details:flow_progress:status

String

Current flow status, typically RUNNING.

details:flow_progress:metrics:num_upserted_rows

Integer

Number of rows inserted or updated since the last event. Delta metric. See Row and byte counters.

details:flow_progress:metrics:num_deleted_rows

Integer

Number of rows deleted since the last event. Delta metric. null for snapshot flows.

details:flow_progress:metrics:num_output_bytes

Integer

Number of compressed bytes uploaded to a volume since the last event. Delta metric. Not reported for query-based connector snapshots. See Query-based connector snapshots.

details:flow_progress:streaming_metrics:discovery_latency_ms

Integer

Time in milliseconds between the source change at event_time:max and when this event was emitted. CDC flows only. Can be null if the source does not expose change timestamps.

details:flow_progress:streaming_metrics:batch_processing_time_ms

Integer

Time in milliseconds the gateway spent reading and uploading the most recent batch of changes. Does not include source-side lag. CDC flows only. Can be null until the first batch completes.

details:flow_progress:streaming_metrics:event_time:max

String

ISO 8601 timestamp of the most recent source change the gateway has read for this table. CDC flows only.

details:operation_progress:type

String

Operation type. CDC_SNAPSHOT for snapshot operations.

details:operation_progress:status

String

Current operation status. IN_PROGRESS while the snapshot is running, COMPLETED when it finishes. Other values include STARTED, CANCELED, and FAILED. Gateways running a query-based connector snapshot also report SCHEDULED while a table is queued. See Query-based connector snapshots.

details:operation_progress:duration_ms

Integer

Total elapsed time of the operation in milliseconds.

details:operation_progress:progress_percent

Double

Snapshot completion percentage (0.0 - 100.0). Cumulative. See Snapshot progress and ETA.

details:operation_progress:estimated_completion_ms

Integer

Estimated time remaining in milliseconds until the snapshot finishes. See Snapshot progress and ETA.

details:operation_progress:cdc_snapshot:target_table_name

String

Fully qualified name of the table being snapshotted.

maturity_level

String

Always STABLE.

Interpret progress metrics​

Row and byte counters​

Applies to num_upserted_rows, num_deleted_rows, and num_output_bytes.

  • Each value is a delta. It represents changes since the previous event, not a cumulative total. To calculate totals, sum the values across events. See Row count queries.
  • Events are emitted even when no data changes occur. Zero values serve as liveness signals that confirm the pipeline is running.
  • num_deleted_rows is null for snapshot flows because snapshots do not produce deletes.
  • Counters reset when the pipeline restarts, refreshes, or resumes. Events emitted before the restart remain in the event log.

Snapshot progress and ETA​

Applies to progress_percent and estimated_completion_ms.

  • progress_percent is cumulative. It accumulates from 0.0 to 100.0 over the lifetime of a snapshot.
  • progress_percent is calculated as (completed_chunks / total_chunks) × 100, so it is approximate, not an exact row-level percentage. Tables that are not split into multiple chunks (typically smaller tables) jump from 0.0 directly to 100.0. Large tables that are split into many chunks update gradually as each chunk completes.
  • A COMPLETED status reports progress_percent = 100.0.
  • A pipeline restart can reset progress metrics, including progress_percent.
  • estimated_completion_ms reflects the estimate at the moment the event was emitted. It decreases as the snapshot progresses and reaches 0 on completion. It can briefly increase if progress slows, and is null until enough data has been processed to produce an estimate.

CDC latency​

Applies to event_time:max, discovery_latency_ms, and batch_processing_time_ms. These metrics are reported for CDC flows only.

  • Each value reflects the state of the metric at the moment the event was emitted.
  • These are ingestion gateway latencies. They measure the path from the source database through the gateway to the Unity Catalog volume. They do not include downstream latency from the Unity Catalog volume to the destination table, which is reported separately in the ingestion pipeline's event log.
  • To estimate source-side lag, subtract batch_processing_time_ms from discovery_latency_ms.
  • discovery_latency_ms and batch_processing_time_ms are reported as 0 if the result would otherwise be negative.

The relationship between these metrics tells you where on the gateway path the latency is concentrated:

Pattern

What it means

Both discovery_latency_ms and batch_processing_time_ms small

CDC is running and fresh.

discovery_latency_ms high, batch_processing_time_ms low

Source-side lag. The gateway is reading promptly. Commits arrived late from the source (replication delay, source log backlog, long-running source transactions).

Both high

Gateway-side lag. The upload pipeline is the bottleneck. Check gateway compute resources and the network path to the Unity Catalog volume.

discovery_latency_ms rising on one table only

Per-table source issue (DDL in flight, lock contention, blocked replication slot, schema change).

discovery_latency_ms rising on every table

Gateway-wide issue (resources, volume connectivity, source CDC log backlog affecting all captures).

event_time:max not advancing across multiple emissions

Source is idle, or the gateway has lost source-database connectivity. Verify source CDC enablement and gateway logs.

Pattern

What it means

Both discovery_latency_ms and batch_processing_time_ms small

CDC is running and fresh.

discovery_latency_ms high, batch_processing_time_ms low

Source-side lag. The gateway is reading promptly. Commits arrived late from the source (replication delay, source log backlog, long-running source transactions).

Both high

Gateway-side lag. The upload pipeline is the bottleneck. Check gateway compute resources and the network path to the Unity Catalog volume.

discovery_latency_ms rising on one table only

Per-table source issue (DDL in flight, lock contention, blocked replication slot, schema change).

discovery_latency_ms rising on every table

Gateway-wide issue (resources, volume connectivity, source CDC log backlog affecting all captures).

event_time:max not advancing across multiple emissions

Source is idle, or the gateway has lost source-database connectivity. Verify source CDC enablement and gateway logs.

Query-based connector snapshots​

When the gateway runs the initial snapshot using the query-based connector (QBC) method, it emits the same snapshot metrics described on this page: flow_progress events for row counts, and operation_progress events for the progress percentage and estimated time remaining (estimated_completion_ms).

These metrics come only from the ingestion gateway. Query-based connector pipelines that run without a gateway (pipelines that read the source directly through a connection) do not emit them, so their event logs contain no snapshot progress events.

A QBC snapshot differs from gateway snapshots (legacy) in the following ways:

  • Output bytes are not reported. flow_progress events report row counts (num_upserted_rows), but not num_output_bytes. Queries that read output bytes, such as the ones in Output bytes queries, return no data for a QBC snapshot. Use the row-count metric to track volume instead.
  • Tables report a SCHEDULED status. Each table reports SCHEDULED while it is queued, before its snapshot starts. Gateway snapshots (legacy) do not report a queued status.
  • Some operation_progress fields are not populated. The duration_ms field and the cdc_snapshot.snapshot_reason field are not set for a QBC snapshot.

To monitor a QBC snapshot, use the snapshot progress queries in Snapshot progress queries, including the Snapshot ETA per table query for estimated time remaining.

Adjust the queries for how a QBC snapshot reports queued tables. The queries in that section identify queued tables by a progress percentage of 0, but a QBC snapshot reports queued tables with a SCHEDULED status and no progress percentage. To classify or count them correctly, change these conditions to check the status:

  • Replace a progress_pct = 0 condition with status = 'SCHEDULED'.
  • To include queued tables in the Snapshot ETA per table query, change its status = 'IN_PROGRESS' filter to status IN ('IN_PROGRESS', 'SCHEDULED').

Sample queries​

The following sample queries show how to monitor your gateway using row counts, output bytes, snapshot progress and ETA, and CDC latency metrics. Replace <pipeline-id> with your gateway ID in each query.

tip

The Ingestion Gateway Progress Monitor notebook contains all queries in this section. Import the notebook into the workspace where your gateway runs, and specify your gateway ID. Run the notebook to inspect snapshot progress, CDC latency, and row counts. You can also adjust the SLA thresholds to match your monitoring requirements.

Row count queries​

Volume per table (last 24 hours)​

Total upserts and deletes for each table over the last 24 hours, with the ingestion phase classified from the flow name suffix. Use this as a dashboard headline to see which tables moved the most data.

SQL
WITH row_events AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
details:flow_progress:metrics:num_upserted_rows::bigint AS upserts,
details:flow_progress:metrics:num_deleted_rows::bigint AS deletes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
)
SELECT
flow_name,
phase,
SUM(upserts) AS rows_upserted_24h,
SUM(COALESCE(deletes, 0)) AS rows_deleted_24h,
SUM(upserts) + SUM(COALESCE(deletes, 0)) AS total_rows_moved_24h
FROM row_events
GROUP BY flow_name, phase
ORDER BY total_rows_moved_24h DESC

Identify silent or stuck tables​

Tables emitting events but reporting zero upserts and zero deletes for the last 60 minutes are candidates for the "stuck" status. Investigate further if the source should be changing. CDC tables that are genuinely idle (for example, overnight) also appear here.

SQL
WITH recent_window AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
COUNT(*) AS emissions_in_window,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS upserts_in_window,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0)) AS deletes_in_window,
MAX(timestamp) AS last_event
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 60 MINUTES
GROUP BY origin.flow_name
)
SELECT
flow_name,
phase,
emissions_in_window,
upserts_in_window,
deletes_in_window,
last_event,
ROUND(TIMESTAMPDIFF(MINUTE, last_event, current_timestamp()), 0) AS minutes_since_last_event
FROM recent_window
WHERE upserts_in_window = 0
AND deletes_in_window = 0
ORDER BY minutes_since_last_event DESC

Per-table timeline with cumulative totals​

Full event-by-event history for a single flow over the last 24 hours, with cumulative row counts. Replace <flow-pattern> with a SQL LIKE pattern (for example, '%customers%_cdc_flow').

SQL
SELECT
origin.flow_name AS flow_name,
origin.update_id AS update_id,
timestamp,
details:flow_progress:metrics:num_upserted_rows::bigint AS upserts_this_period,
COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0) AS deletes_this_period,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint)
OVER (PARTITION BY origin.flow_name, origin.update_id
ORDER BY timestamp
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_upserts_this_run,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0))
OVER (PARTITION BY origin.flow_name, origin.update_id
ORDER BY timestamp
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_deletes_this_run
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name LIKE '<flow-pattern>'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
ORDER BY timestamp

Output bytes queries​

Bytes per table (last 24 hours)​

Total bytes uploaded to a volume per table over the last 24 hours, with human-friendly MB and GB units. Sort descending to see the heaviest-volume tables.

SQL
WITH byte_events AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
details:flow_progress:metrics:num_output_bytes::bigint AS output_bytes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
)
SELECT
flow_name,
phase,
SUM(output_bytes) AS bytes_24h,
ROUND(SUM(output_bytes) / 1024.0 / 1024.0, 2) AS mb_24h,
ROUND(SUM(output_bytes) / 1024.0 / 1024.0 / 1024.0, 3) AS gb_24h
FROM byte_events
GROUP BY flow_name, phase
ORDER BY bytes_24h DESC

Throughput trend (MB per minute)​

Per-minute time series of bytes uploaded across all flows over the last 24 hours. Render as a line chart to spot throughput patterns and stalls.

SQL
SELECT
DATE_TRUNC('MINUTE', timestamp) AS ts_minute,
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
ROUND(SUM(details:flow_progress:metrics:num_output_bytes::bigint) / 1024.0 / 1024.0, 2) AS mb_per_minute
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY DATE_TRUNC('MINUTE', timestamp), origin.flow_name
ORDER BY origin.flow_name, ts_minute

Average bytes per row (wide-table or LOB detector)​

Joins num_output_bytes with row counts to compute average bytes per row per table. High values usually indicate LOB or wide-schema tables that drive throughput cost. Useful for capacity planning and schema review.

SQL
WITH joined AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
SUM(details:flow_progress:metrics:num_output_bytes::bigint) AS total_bytes,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS total_upserts,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0)) AS total_deletes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY origin.flow_name
)
SELECT
flow_name,
phase,
total_upserts + total_deletes AS total_rows,
ROUND(total_bytes / 1024.0 / 1024.0, 2) AS total_mb,
ROUND(total_bytes / NULLIF(total_upserts + total_deletes, 0), 0) AS avg_bytes_per_row
FROM joined
WHERE total_bytes > 0
ORDER BY avg_bytes_per_row DESC

Snapshot progress queries​

Overall snapshot progress​

The following query returns a single-row summary of how many tables are completed, in progress, or queued. Results reflect the latest reported state per table across all pipeline updates in the event-log retention window, so tables completed in a prior update continue to count toward tables_completed after a refresh or restart.

SQL
WITH latest_per_table AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
)
SELECT
COUNT(*) AS total_tables,
SUM(CASE WHEN status = 'COMPLETED' THEN 1 ELSE 0 END) AS tables_completed,
SUM(CASE WHEN status = 'IN_PROGRESS' AND progress_pct > 0 AND progress_pct < 100 THEN 1 ELSE 0 END) AS tables_in_progress,
SUM(CASE WHEN progress_pct = 0 THEN 1 ELSE 0 END) AS tables_not_started,
ROUND(AVG(progress_pct), 2) AS overall_progress_pct,
ROUND(SUM(CASE WHEN status = 'COMPLETED' THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 1) AS pct_tables_done
FROM latest_per_table
WHERE rn = 1

Per-table snapshot status board​

The following query returns each table in the pipeline with its latest reported status and progress percentage. Results include tables completed in prior updates, so no tables appear as missing rows.

SQL
WITH table_status AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
timestamp,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
)
SELECT
flow_name,
status,
ROUND(progress_pct, 2) AS progress_pct,
timestamp AS last_update,
CASE
WHEN status = 'COMPLETED' THEN 'Done'
WHEN progress_pct = 0 THEN 'Queued'
ELSE 'Active'
END AS phase
FROM table_status
WHERE rn = 1
ORDER BY
CASE status WHEN 'IN_PROGRESS' THEN 0 WHEN 'COMPLETED' THEN 1 ELSE 2 END,
progress_pct ASC

Snapshot rows and bytes loaded in the current update​

The following query combines progress percentage, rows upserted, and bytes uploaded for each table in the current snapshot run.

SQL
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
progress AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:status::string AS status,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
),
volume AS (
SELECT
origin.flow_name AS flow_name,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS rows_loaded,
SUM(details:flow_progress:metrics:num_output_bytes::bigint) AS bytes_loaded
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_snapshot_flow'
AND origin.update_id = (SELECT update_id FROM latest_update)
GROUP BY origin.flow_name
)
SELECT
p.flow_name,
p.status,
ROUND(p.progress_pct, 2) AS progress_pct,
v.rows_loaded,
ROUND(v.bytes_loaded / 1024.0 / 1024.0, 2) AS mb_loaded,
ROUND(v.bytes_loaded / 1024.0 / 1024.0 / 1024.0, 3) AS gb_loaded
FROM progress p
LEFT JOIN volume v ON p.flow_name = v.flow_name
WHERE p.rn = 1
ORDER BY p.progress_pct ASC

Stalled snapshot detection​

The following query returns snapshot tables whose progress_percent has not changed in the last 30 minutes. Use it to identify snapshots that have stalled but are still active.

SQL
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
recent AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:status::string AS status,
timestamp
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
AND timestamp >= current_timestamp() - INTERVAL 30 MINUTES
)
SELECT
flow_name,
ROUND(MIN(progress_pct), 2) AS min_pct_30min,
ROUND(MAX(progress_pct), 2) AS max_pct_30min,
ROUND(MAX(progress_pct) - MIN(progress_pct), 2) AS pct_change_30min,
COUNT(*) AS events_in_window,
MAX(timestamp) AS last_event_ts
FROM recent
WHERE status = 'IN_PROGRESS'
GROUP BY flow_name
HAVING MAX(progress_pct) - MIN(progress_pct) = 0
AND MAX(progress_pct) < 100
ORDER BY max_pct_30min ASC

Snapshot ETA per table​

The following query returns the current progress percentage and estimated time to completion for each table in the active snapshot run. Only tables with a status of IN_PROGRESS are returned. To include completed and queued tables, remove the status = 'IN_PROGRESS' filter.

SQL
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
latest_per_flow AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:estimated_completion_ms::bigint AS eta_ms,
timestamp,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
)
SELECT
flow_name,
status,
ROUND(progress_pct, 2) AS progress_pct,
eta_ms,
ROUND(eta_ms / 1000.0 / 60.0, 1) AS eta_minutes,
ROUND(eta_ms / 1000.0 / 3600.0, 2) AS eta_hours,
timestamp AS last_update
FROM latest_per_flow
WHERE rn = 1
AND status = 'IN_PROGRESS'
ORDER BY eta_ms DESC NULLS LAST

CDC latency queries​

To interpret the values that these queries return, see CDC latency. These values measure latency on the gateway path. To see how current each destination table is, see Monitor data freshness for change data capture (CDC) tables.

CDC latency per table​

The following query returns the last 30 minutes of CDC latency events for each CDC table. With events emitted every five minutes, this produces about six rows per table. Results are sorted by discovery_latency_ms so the most lagged tables appear first.

SQL
SELECT
origin.flow_name AS flow_name,
details:flow_progress:streaming_metrics:event_time:max::string AS latest_source_commit_seen,
details:flow_progress:streaming_metrics:discovery_latency_ms::bigint AS discovery_latency_ms,
details:flow_progress:streaming_metrics:batch_processing_time_ms::bigint AS batch_processing_time_ms,
timestamp AS emission_ts
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_cdc_flow'
AND details:flow_progress:streaming_metrics IS NOT NULL
AND timestamp >= current_timestamp() - INTERVAL 30 MINUTES
ORDER BY discovery_latency_ms DESC NULLS LAST, flow_name, emission_ts DESC

CDC latency time series​

The following query returns the hourly average of discovery_latency_ms and batch_processing_time_ms for each CDC flow in the last 24 hours. Use it to identify when freshness degraded. For pipelines with many CDC flows, filter by flow name to limit the result set.

SQL
SELECT
DATE_TRUNC('HOUR', timestamp) AS ts_hour,
origin.flow_name AS flow_name,
ROUND(AVG(details:flow_progress:streaming_metrics:discovery_latency_ms::bigint) / 1000.0, 2) AS avg_discovery_latency_sec,
ROUND(AVG(details:flow_progress:streaming_metrics:batch_processing_time_ms::bigint) / 1000.0, 2) AS avg_batch_processing_sec
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_cdc_flow'
AND details:flow_progress:streaming_metrics IS NOT NULL
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY DATE_TRUNC('HOUR', timestamp), origin.flow_name
ORDER BY origin.flow_name, ts_hour

Troubleshooting​

No progress events appear​

If you don't see progress events in the event log:

  1. Wait at least five minutes after the pipeline starts.
  2. Check that the pipeline is actively running and ingesting.
  3. Check that your query filters on level = 'METRICS' and the correct event type (flow_progress or operation_progress).
  4. If events are still missing, or they arrive at an interval other than five minutes, contact Databricks support.

Metrics show zero after pipeline restart​

A pipeline restart, refresh, or resume can reset progress metrics. The gateway starts reporting new values immediately, and events emitted before the restart remain in the event log. See Interpret progress metrics.

Missing metrics for some tables​

If some tables don't show progress events:

  1. Confirm that the table is included in the gateway configuration and is not filtered out.
  2. For the CDC phase, make sure that the source table has CDC or change tracking enabled.
  3. Note that operation_progress events are emitted only for snapshot flows. CDC flows do not emit operation_progress events because CDC has no concept of completion.

Snapshot ETA or CDC latency fields are null​

Some fields are null until the gateway has enough information to report them. For when each field is null, see Event fields and Interpret progress metrics.

Additional resources​