Monitor ingestion gateway progress with event logs
Applies to: SaaS connectors
Database connectors
Learn how to use event logs to monitor the progress of ingestion gateways in real time. The gateway emits per-table metrics for both the snapshot and change data capture (CDC) phases, so you can track pipeline health, identify stalled tables, and build alerts and dashboards from structured event data.
Progress events allow you to:
- Track how many rows and bytes have been ingested per table without waiting for pipeline completion.
- Monitor snapshot progress and estimated time to completion for large initial loads.
- Measure CDC latency from the source commit to the Unity Catalog volume for each table.
- Confirm that the pipeline is actively running, even when no data changes occur.
How progress events work
The gateway emits the following event types every five minutes for each table in your pipeline. The first events appear within five minutes of the pipeline starting.
flow_progress: Emitted for both snapshot and CDC flows. Reports rows upserted, rows deleted (CDC only), and output bytes since the previous event. For CDC flows, also reports latency metrics.operation_progress: Emitted for snapshot flows only. Reports the snapshot completion percentage and the estimated time remaining until the snapshot finishes.
Events are emitted for every table, including tables with no changes since the previous event. A table that reports zero rows is idle. A table that reports no events hasn't been picked up by the pipeline.
Both event types use the METRICS log level. You can query them from the event log table using SQL.
Access progress events
Progress events are stored in the event log table. To access them:
- Navigate to your gateway in the Databricks workspace.
- Click the Event log tab to view events in the UI.
- Query the event log table directly using SQL for detailed analysis.
Query the event log table
To query flow_progress events for row and byte counters:
SELECT
timestamp,
CONCAT(origin.catalog_name, '.', origin.schema_name, '.', origin.dataset_name) AS table_name,
details:flow_progress:metrics:num_upserted_rows::bigint AS rows_upserted,
COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0) AS rows_deleted,
details:flow_progress:metrics:num_output_bytes::bigint AS output_bytes,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'snapshot'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'cdc'
ELSE 'unknown'
END AS ingestion_phase
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.pipeline_type = 'INGESTION_GATEWAY'
ORDER BY timestamp DESC
To query operation_progress events for snapshot progress percentage:
SELECT
timestamp,
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.pipeline_type = 'INGESTION_GATEWAY'
ORDER BY timestamp DESC
Replace <pipeline-id> with your gateway ID.
Understand the event structure
The following examples show the JSON structure for each event type:
Snapshot flow progress event structure
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:14.175Z",
"level": "METRICS",
"event_type": "flow_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_snapshot_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Completed a streaming update of 'main.sales.customers_snapshot_flow'.",
"details": {
"flow_progress": {
"status": "RUNNING",
"metrics": {
"num_upserted_rows": 7512704,
"num_deleted_rows": null,
"num_output_bytes": 458752000
}
}
},
"maturity_level": "STABLE"
}
CDC flow progress event structure
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:57.426Z",
"level": "METRICS",
"event_type": "flow_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_cdc_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Completed a streaming update of 'main.sales.customers_cdc_flow'.",
"details": {
"flow_progress": {
"status": "RUNNING",
"metrics": {
"num_upserted_rows": 25,
"num_deleted_rows": 3,
"num_output_bytes": 18432
},
"streaming_metrics": {
"discovery_latency_ms": 12450,
"batch_processing_time_ms": 8100,
"event_time": {
"max": "2025-10-14T13:33:45.000Z"
}
}
}
},
"maturity_level": "STABLE"
}
Snapshot operation progress event structure
{
"id": "01234567-89ab-cdef-0123-456789abcdef",
"timestamp": "2025-10-14T13:33:14.175Z",
"level": "METRICS",
"event_type": "operation_progress",
"origin": {
"pipeline_type": "INGESTION_GATEWAY",
"pipeline_name": "MyPipeline",
"dataset_name": "customers",
"catalog_name": "main",
"schema_name": "sales",
"flow_name": "main.sales.customers_snapshot_flow",
"ingestion_source_type": "SQLSERVER"
},
"message": "Snapshot in progress for 'main.sales.customers'.",
"details": {
"operation_progress": {
"type": "CDC_SNAPSHOT",
"status": "IN_PROGRESS",
"duration_ms": 3600000,
"progress_percent": 65.5,
"estimated_completion_ms": 1885000,
"cdc_snapshot": {
"target_table_name": "main.sales.customers",
"snapshot_timestamp": 1737542400000,
"snapshot_reason": "NEW_TABLE"
}
}
},
"maturity_level": "STABLE"
}
Event fields
The following table describes the key fields in progress events:
Field | Type | Description |
|---|---|---|
| String | Either |
| String | Always |
| String | ISO 8601 timestamp when the event was generated. |
| String | Always |
| String | Name of the gateway. |
| String | Name of the table being ingested. |
| String | Unity Catalog catalog name. |
| String | Unity Catalog schema name. |
| String | Flow identifier that indicates the ingestion phase. Format: |
| String | Source database type (for example, |
| String | Current flow status, typically |
| Integer | Number of rows inserted or updated since the last event. Delta metric. See Row and byte counters. |
| Integer | Number of rows deleted since the last event. Delta metric. |
| Integer | Number of compressed bytes uploaded to a volume since the last event. Delta metric. Not reported for query-based connector snapshots. See Query-based connector snapshots. |
| Integer | Time in milliseconds between the source change at |
| Integer | Time in milliseconds the gateway spent reading and uploading the most recent batch of changes. Does not include source-side lag. CDC flows only. Can be |
| String | ISO 8601 timestamp of the most recent source change the gateway has read for this table. CDC flows only. |
| String | Operation type. |
| String | Current operation status. |
| Integer | Total elapsed time of the operation in milliseconds. |
| Double | Snapshot completion percentage ( |
| Integer | Estimated time remaining in milliseconds until the snapshot finishes. See Snapshot progress and ETA. |
| String | Fully qualified name of the table being snapshotted. |
| String | Always |
Interpret progress metrics
Row and byte counters
Applies to num_upserted_rows, num_deleted_rows, and num_output_bytes.
- Each value is a delta. It represents changes since the previous event, not a cumulative total. To calculate totals, sum the values across events. See Row count queries.
- Events are emitted even when no data changes occur. Zero values serve as liveness signals that confirm the pipeline is running.
num_deleted_rowsisnullfor snapshot flows because snapshots do not produce deletes.- Counters reset when the pipeline restarts, refreshes, or resumes. Events emitted before the restart remain in the event log.
Snapshot progress and ETA
Applies to progress_percent and estimated_completion_ms.
progress_percentis cumulative. It accumulates from0.0to100.0over the lifetime of a snapshot.progress_percentis calculated as(completed_chunks / total_chunks) × 100, so it is approximate, not an exact row-level percentage. Tables that are not split into multiple chunks (typically smaller tables) jump from0.0directly to100.0. Large tables that are split into many chunks update gradually as each chunk completes.- A
COMPLETEDstatus reportsprogress_percent = 100.0. - A pipeline restart can reset progress metrics, including
progress_percent. estimated_completion_msreflects the estimate at the moment the event was emitted. It decreases as the snapshot progresses and reaches0on completion. It can briefly increase if progress slows, and isnulluntil enough data has been processed to produce an estimate.
CDC latency
Applies to event_time:max, discovery_latency_ms, and batch_processing_time_ms. These metrics are reported for CDC flows only.
- Each value reflects the state of the metric at the moment the event was emitted.
- These are ingestion gateway latencies. They measure the path from the source database through the gateway to the Unity Catalog volume. They do not include downstream latency from the Unity Catalog volume to the destination table, which is reported separately in the ingestion pipeline's event log.
- To estimate source-side lag, subtract
batch_processing_time_msfromdiscovery_latency_ms. discovery_latency_msandbatch_processing_time_msare reported as0if the result would otherwise be negative.
The relationship between these metrics tells you where on the gateway path the latency is concentrated:
Pattern | What it means |
|---|---|
Both | CDC is running and fresh. |
| Source-side lag. The gateway is reading promptly. Commits arrived late from the source (replication delay, source log backlog, long-running source transactions). |
Both high | Gateway-side lag. The upload pipeline is the bottleneck. Check gateway compute resources and the network path to the Unity Catalog volume. |
| Per-table source issue (DDL in flight, lock contention, blocked replication slot, schema change). |
| Gateway-wide issue (resources, volume connectivity, source CDC log backlog affecting all captures). |
| Source is idle, or the gateway has lost source-database connectivity. Verify source CDC enablement and gateway logs. |
Query-based connector snapshots
When the gateway runs the initial snapshot using the query-based connector (QBC) method, it emits the same snapshot metrics described on this page: flow_progress events for row counts, and operation_progress events for the progress percentage and estimated time remaining (estimated_completion_ms).
These metrics come only from the ingestion gateway. Query-based connector pipelines that run without a gateway (pipelines that read the source directly through a connection) do not emit them, so their event logs contain no snapshot progress events.
A QBC snapshot differs from gateway snapshots (legacy) in the following ways:
- Output bytes are not reported.
flow_progressevents report row counts (num_upserted_rows), but notnum_output_bytes. Queries that read output bytes, such as the ones in Output bytes queries, return no data for a QBC snapshot. Use the row-count metric to track volume instead. - Tables report a
SCHEDULEDstatus. Each table reportsSCHEDULEDwhile it is queued, before its snapshot starts. Gateway snapshots (legacy) do not report a queued status. - Some
operation_progressfields are not populated. Theduration_msfield and thecdc_snapshot.snapshot_reasonfield are not set for a QBC snapshot.
To monitor a QBC snapshot, use the snapshot progress queries in Snapshot progress queries, including the Snapshot ETA per table query for estimated time remaining.
Adjust the queries for how a QBC snapshot reports queued tables. The queries in that section identify queued tables by a progress percentage of 0, but a QBC snapshot reports queued tables with a SCHEDULED status and no progress percentage. To classify or count them correctly, change these conditions to check the status:
- Replace a
progress_pct = 0condition withstatus = 'SCHEDULED'. - To include queued tables in the Snapshot ETA per table query, change its
status = 'IN_PROGRESS'filter tostatus IN ('IN_PROGRESS', 'SCHEDULED').
Sample queries
The following sample queries show how to monitor your gateway using row counts, output bytes, snapshot progress and ETA, and CDC latency metrics. Replace <pipeline-id> with your gateway ID in each query.
The Ingestion Gateway Progress Monitor notebook contains all queries in this section. Import the notebook into the workspace where your gateway runs, and specify your gateway ID. Run the notebook to inspect snapshot progress, CDC latency, and row counts. You can also adjust the SLA thresholds to match your monitoring requirements.
Row count queries
Volume per table (last 24 hours)
Total upserts and deletes for each table over the last 24 hours, with the ingestion phase classified from the flow name suffix. Use this as a dashboard headline to see which tables moved the most data.
WITH row_events AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
details:flow_progress:metrics:num_upserted_rows::bigint AS upserts,
details:flow_progress:metrics:num_deleted_rows::bigint AS deletes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
)
SELECT
flow_name,
phase,
SUM(upserts) AS rows_upserted_24h,
SUM(COALESCE(deletes, 0)) AS rows_deleted_24h,
SUM(upserts) + SUM(COALESCE(deletes, 0)) AS total_rows_moved_24h
FROM row_events
GROUP BY flow_name, phase
ORDER BY total_rows_moved_24h DESC
Identify silent or stuck tables
Tables emitting events but reporting zero upserts and zero deletes for the last 60 minutes are candidates for the "stuck" status. Investigate further if the source should be changing. CDC tables that are genuinely idle (for example, overnight) also appear here.
WITH recent_window AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
COUNT(*) AS emissions_in_window,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS upserts_in_window,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0)) AS deletes_in_window,
MAX(timestamp) AS last_event
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 60 MINUTES
GROUP BY origin.flow_name
)
SELECT
flow_name,
phase,
emissions_in_window,
upserts_in_window,
deletes_in_window,
last_event,
ROUND(TIMESTAMPDIFF(MINUTE, last_event, current_timestamp()), 0) AS minutes_since_last_event
FROM recent_window
WHERE upserts_in_window = 0
AND deletes_in_window = 0
ORDER BY minutes_since_last_event DESC
Per-table timeline with cumulative totals
Full event-by-event history for a single flow over the last 24 hours, with cumulative row counts. Replace <flow-pattern> with a SQL LIKE pattern (for example, '%customers%_cdc_flow').
SELECT
origin.flow_name AS flow_name,
origin.update_id AS update_id,
timestamp,
details:flow_progress:metrics:num_upserted_rows::bigint AS upserts_this_period,
COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0) AS deletes_this_period,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint)
OVER (PARTITION BY origin.flow_name, origin.update_id
ORDER BY timestamp
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_upserts_this_run,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0))
OVER (PARTITION BY origin.flow_name, origin.update_id
ORDER BY timestamp
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cumulative_deletes_this_run
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name LIKE '<flow-pattern>'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
ORDER BY timestamp
Output bytes queries
Bytes per table (last 24 hours)
Total bytes uploaded to a volume per table over the last 24 hours, with human-friendly MB and GB units. Sort descending to see the heaviest-volume tables.
WITH byte_events AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
details:flow_progress:metrics:num_output_bytes::bigint AS output_bytes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
)
SELECT
flow_name,
phase,
SUM(output_bytes) AS bytes_24h,
ROUND(SUM(output_bytes) / 1024.0 / 1024.0, 2) AS mb_24h,
ROUND(SUM(output_bytes) / 1024.0 / 1024.0 / 1024.0, 3) AS gb_24h
FROM byte_events
GROUP BY flow_name, phase
ORDER BY bytes_24h DESC
Throughput trend (MB per minute)
Per-minute time series of bytes uploaded across all flows over the last 24 hours. Render as a line chart to spot throughput patterns and stalls.
SELECT
DATE_TRUNC('MINUTE', timestamp) AS ts_minute,
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
ROUND(SUM(details:flow_progress:metrics:num_output_bytes::bigint) / 1024.0 / 1024.0, 2) AS mb_per_minute
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY DATE_TRUNC('MINUTE', timestamp), origin.flow_name
ORDER BY origin.flow_name, ts_minute
Average bytes per row (wide-table or LOB detector)
Joins num_output_bytes with row counts to compute average bytes per row per table. High values usually indicate LOB or wide-schema tables that drive throughput cost. Useful for capacity planning and schema review.
WITH joined AS (
SELECT
origin.flow_name AS flow_name,
CASE
WHEN origin.flow_name LIKE '%_snapshot_flow' THEN 'SNAPSHOT'
WHEN origin.flow_name LIKE '%_cdc_flow' THEN 'CDC'
ELSE 'OTHER'
END AS phase,
SUM(details:flow_progress:metrics:num_output_bytes::bigint) AS total_bytes,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS total_upserts,
SUM(COALESCE(details:flow_progress:metrics:num_deleted_rows::bigint, 0)) AS total_deletes
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY origin.flow_name
)
SELECT
flow_name,
phase,
total_upserts + total_deletes AS total_rows,
ROUND(total_bytes / 1024.0 / 1024.0, 2) AS total_mb,
ROUND(total_bytes / NULLIF(total_upserts + total_deletes, 0), 0) AS avg_bytes_per_row
FROM joined
WHERE total_bytes > 0
ORDER BY avg_bytes_per_row DESC
Snapshot progress queries
Overall snapshot progress
The following query returns a single-row summary of how many tables are completed, in progress, or queued. Results reflect the latest reported state per table across all pipeline updates in the event-log retention window, so tables completed in a prior update continue to count toward tables_completed after a refresh or restart.
WITH latest_per_table AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
)
SELECT
COUNT(*) AS total_tables,
SUM(CASE WHEN status = 'COMPLETED' THEN 1 ELSE 0 END) AS tables_completed,
SUM(CASE WHEN status = 'IN_PROGRESS' AND progress_pct > 0 AND progress_pct < 100 THEN 1 ELSE 0 END) AS tables_in_progress,
SUM(CASE WHEN progress_pct = 0 THEN 1 ELSE 0 END) AS tables_not_started,
ROUND(AVG(progress_pct), 2) AS overall_progress_pct,
ROUND(SUM(CASE WHEN status = 'COMPLETED' THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 1) AS pct_tables_done
FROM latest_per_table
WHERE rn = 1
Per-table snapshot status board
The following query returns each table in the pipeline with its latest reported status and progress percentage. Results include tables completed in prior updates, so no tables appear as missing rows.
WITH table_status AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
timestamp,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
)
SELECT
flow_name,
status,
ROUND(progress_pct, 2) AS progress_pct,
timestamp AS last_update,
CASE
WHEN status = 'COMPLETED' THEN 'Done'
WHEN progress_pct = 0 THEN 'Queued'
ELSE 'Active'
END AS phase
FROM table_status
WHERE rn = 1
ORDER BY
CASE status WHEN 'IN_PROGRESS' THEN 0 WHEN 'COMPLETED' THEN 1 ELSE 2 END,
progress_pct ASC
Snapshot rows and bytes loaded in the current update
The following query combines progress percentage, rows upserted, and bytes uploaded for each table in the current snapshot run.
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
progress AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:status::string AS status,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
),
volume AS (
SELECT
origin.flow_name AS flow_name,
SUM(details:flow_progress:metrics:num_upserted_rows::bigint) AS rows_loaded,
SUM(details:flow_progress:metrics:num_output_bytes::bigint) AS bytes_loaded
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_snapshot_flow'
AND origin.update_id = (SELECT update_id FROM latest_update)
GROUP BY origin.flow_name
)
SELECT
p.flow_name,
p.status,
ROUND(p.progress_pct, 2) AS progress_pct,
v.rows_loaded,
ROUND(v.bytes_loaded / 1024.0 / 1024.0, 2) AS mb_loaded,
ROUND(v.bytes_loaded / 1024.0 / 1024.0 / 1024.0, 3) AS gb_loaded
FROM progress p
LEFT JOIN volume v ON p.flow_name = v.flow_name
WHERE p.rn = 1
ORDER BY p.progress_pct ASC
Stalled snapshot detection
The following query returns snapshot tables whose progress_percent has not changed in the last 30 minutes. Use it to identify snapshots that have stalled but are still active.
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
recent AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:status::string AS status,
timestamp
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
AND timestamp >= current_timestamp() - INTERVAL 30 MINUTES
)
SELECT
flow_name,
ROUND(MIN(progress_pct), 2) AS min_pct_30min,
ROUND(MAX(progress_pct), 2) AS max_pct_30min,
ROUND(MAX(progress_pct) - MIN(progress_pct), 2) AS pct_change_30min,
COUNT(*) AS events_in_window,
MAX(timestamp) AS last_event_ts
FROM recent
WHERE status = 'IN_PROGRESS'
GROUP BY flow_name
HAVING MAX(progress_pct) - MIN(progress_pct) = 0
AND MAX(progress_pct) < 100
ORDER BY max_pct_30min ASC
Snapshot ETA per table
The following query returns the current progress percentage and estimated time to completion for each table in the active snapshot run. Only tables with a status of IN_PROGRESS are returned. To include completed and queued tables, remove the status = 'IN_PROGRESS' filter.
WITH latest_update AS (
SELECT origin.update_id AS update_id
FROM event_log('<pipeline-id>')
WHERE event_type = 'create_update'
ORDER BY timestamp DESC LIMIT 1
),
latest_per_flow AS (
SELECT
origin.flow_name AS flow_name,
details:operation_progress:status::string AS status,
details:operation_progress:progress_percent::double AS progress_pct,
details:operation_progress:estimated_completion_ms::bigint AS eta_ms,
timestamp,
ROW_NUMBER() OVER (PARTITION BY origin.flow_name ORDER BY timestamp DESC) AS rn
FROM event_log('<pipeline-id>')
WHERE event_type = 'operation_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.update_id = (SELECT update_id FROM latest_update)
)
SELECT
flow_name,
status,
ROUND(progress_pct, 2) AS progress_pct,
eta_ms,
ROUND(eta_ms / 1000.0 / 60.0, 1) AS eta_minutes,
ROUND(eta_ms / 1000.0 / 3600.0, 2) AS eta_hours,
timestamp AS last_update
FROM latest_per_flow
WHERE rn = 1
AND status = 'IN_PROGRESS'
ORDER BY eta_ms DESC NULLS LAST
CDC latency queries
To interpret the values that these queries return, see CDC latency. These values measure latency on the gateway path. To see how current each destination table is, see Monitor data freshness for change data capture (CDC) tables.
CDC latency per table
The following query returns the last 30 minutes of CDC latency events for each CDC table. With events emitted every five minutes, this produces about six rows per table. Results are sorted by discovery_latency_ms so the most lagged tables appear first.
SELECT
origin.flow_name AS flow_name,
details:flow_progress:streaming_metrics:event_time:max::string AS latest_source_commit_seen,
details:flow_progress:streaming_metrics:discovery_latency_ms::bigint AS discovery_latency_ms,
details:flow_progress:streaming_metrics:batch_processing_time_ms::bigint AS batch_processing_time_ms,
timestamp AS emission_ts
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_cdc_flow'
AND details:flow_progress:streaming_metrics IS NOT NULL
AND timestamp >= current_timestamp() - INTERVAL 30 MINUTES
ORDER BY discovery_latency_ms DESC NULLS LAST, flow_name, emission_ts DESC
CDC latency time series
The following query returns the hourly average of discovery_latency_ms and batch_processing_time_ms for each CDC flow in the last 24 hours. Use it to identify when freshness degraded. For pipelines with many CDC flows, filter by flow name to limit the result set.
SELECT
DATE_TRUNC('HOUR', timestamp) AS ts_hour,
origin.flow_name AS flow_name,
ROUND(AVG(details:flow_progress:streaming_metrics:discovery_latency_ms::bigint) / 1000.0, 2) AS avg_discovery_latency_sec,
ROUND(AVG(details:flow_progress:streaming_metrics:batch_processing_time_ms::bigint) / 1000.0, 2) AS avg_batch_processing_sec
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
AND level = 'METRICS'
AND origin.flow_name IS NOT NULL
AND origin.pipeline_type = 'INGESTION_GATEWAY'
AND origin.flow_name LIKE '%_cdc_flow'
AND details:flow_progress:streaming_metrics IS NOT NULL
AND timestamp >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY DATE_TRUNC('HOUR', timestamp), origin.flow_name
ORDER BY origin.flow_name, ts_hour
Troubleshooting
No progress events appear
If you don't see progress events in the event log:
- Wait at least five minutes after the pipeline starts.
- Check that the pipeline is actively running and ingesting.
- Check that your query filters on
level = 'METRICS'and the correct event type (flow_progressoroperation_progress). - If events are still missing, or they arrive at an interval other than five minutes, contact Databricks support.
Metrics show zero after pipeline restart
A pipeline restart, refresh, or resume can reset progress metrics. The gateway starts reporting new values immediately, and events emitted before the restart remain in the event log. See Interpret progress metrics.
Missing metrics for some tables
If some tables don't show progress events:
- Confirm that the table is included in the gateway configuration and is not filtered out.
- For the CDC phase, make sure that the source table has CDC or change tracking enabled.
- Note that
operation_progressevents are emitted only for snapshot flows. CDC flows do not emitoperation_progressevents because CDC has no concept of completion.
Snapshot ETA or CDC latency fields are null
Some fields are null until the gateway has enough information to report them. For when each field is null, see Event fields and Interpret progress metrics.