Zerobus Ingest system table reference

This article is a reference for the zerobus system tables, which track Zerobus Ingest activity in your workspace. These tables include your account records from all workspaces in your same region. To see records from another region, you must view the tables from a workspace deployed in that region.

Requirements

• To access these system tables, users must either:
  • Be both a metastore admin and a workspace admin, or
  • Have USE and SELECT permissions on the system schemas. See Grant access to system tables.

Available Zerobus Ingest tables

All Zerobus Ingest related system tables live in the system.lakeflow schema. This includes the following tables:

Table	Description	Supports streaming	Free retention period	Includes global or regional data
Zerobus_stream (Beta)	Tracks all streams created using Zerobus Ingest	Yes	365 days	Regional
Zerobus_ingest (Beta)	Tracks all data ingested into Delta tables using Zerobus Ingest	Yes	365 days	Regional

Zerobus stream table schema reference

The zerobus_stream table stores all data related to stream events incurred by Zerobus Ingest usage. This includes events that occur when a stream is created, when an error occurs during the stream's lifetime, and when a stream is closed.

Table path: system.lakeflow.zerobus_stream

Column name	Data type	Description	Example
account_id	string	The ID of the account responsible for the stream	23e22ba4-87b9-4cc2-9770-d10b894bxx
workspace_id	string	The ID of the workspace responsible for the stream	1234567890123456
stream_id	string	The ID of the stream	ccr4bf53-127a-4487-camr-cab42c2db9b2
event_time	timestamp	The timestamp of when the event occurred	2025-12-19T18:20:26.994+00:00
producer_id	string	The ID of the producer that is responsible for the stream creation	This column is not currently supported.
opened_time	timestamp	The timestamp of when the stream was opened	2025-12-19T18:20:27.209+00:00
closed_time	timestamp	The timestamp of when the stream was closed	2025-12-19T18:20:26.994+00:00
table_id	string	The ID of the table that is being ingested into	0c5ca387-5v3c-4308-90ad-b91dedb22e32
table_name	string	The fully qualified name of the table that is being ingested into	example_table
protocol	string	The protocol used to ingest data. Possible values are GRPC and HTTP.	GRPC
data_format	string	The format of data used on the stream. Possible values are PROTOBUF and JSON.	PROTOBUF
errors	array	Array of errors that have occurred during the stream's lifetime	See Error reference

Example queries for the stream table

The following query returns the number of currently opened streams for a specified table:

SQL

SELECT COUNT(stream.stream_id)
FROM system.lakeflow.zerobus_stream AS stream
WHERE stream.table_id=:table_id AND stream.closed_time IS NULL AND stream.opened_time > CURRENT_TIMESTAMP - INTERVAL '15 minutes'

The following query returns the IDs of all streams that incurred an error:

SQL

SELECT DISTINCT(stream.stream_id)
FROM system.lakeflow.zerobus_stream AS stream
WHERE size(stream.errors) > 0

Zerobus ingest table schema reference

The zerobus_ingest table stores all data related to records ingested using Zerobus Ingest. This data is not record-level granular. The data is aggregated by batches written to the Delta table, identified using the commit_version.

Table path: system.lakeflow.zerobus_ingest

Column name	Data type	Description	Example
account_id	string	ID of the account that owns the given workspace	23e22ba4-87b9-4cc2-9770-d10b894bxx
workspace_id	string	ID of the workspace that the stream is created in	1234567890123456
stream_id	string	ID of the stream that performed the ingestion	ccr4bf53-127a-4487-camr-cab42c2db9b2
commit_version	bigint	Unique version number of the commit to the Delta	15104
table_id	string	ID of the table that is being written into	0c5ca387-5v3c-4308-90ad-b91dedb22e32
table_name	string	Fully qualified table name in human readable format	example_table
commit_time	timestamp	Timestamp of the commit	2025-12-19T18:20:27.209+00:00
committed_bytes	bigint	Size of the ingested data in bytes	3532498
committed_records	bigint	Number of committed records	3445
tags	array	Custom tags provided by Zerobus Ingest that can be used for tracking usage origin and other metadata	["DIRECT_WRITE"]
errors	array	Array of errors that have occurred during the ingestion	See Error reference

Example queries for the ingestion table

The following query gets the average insertion rate for every table during a specified time range:

SQL

SELECT ingest.table_id AS table_id, SUM(ingest.committed_bytes) / (SECONDS(:end_timestamp - :start_timestamp)) AS average_insertion_rate
FROM system.lakeflow.zerobus_ingest AS ingest
WHERE ingest.commit_time >= :start_timestamp AND ingest.commit_time <= :end_timestamp
GROUP BY ingest.table_id

The following query gets the total amount of data and records ingested into a table during a specified time range:

SQL

SELECT SUM(ingest.committed_bytes) as 'Total data', SUM(ingest.committed_records) as 'Total records'
FROM system.lakeflow.zerobus_ingest AS ingest
WHERE ingest.commit_time >= :start_timestamp AND ingest.commit_time <= :end_timestamp AND ingest.table_name = :table_name

Error reference

Both the stream and ingest tables include an errors column, which includes an array of errors that occurred during the stream or ingest, with each element representing a separate error. These array elements are recorded as objects with the following fields:

Field name	Data type	Description
error_code	bigint	Error code
error_message	string	Full error message
timestamp	timestamp	Timestamp of when the error occurred

Common join patterns

The following query shows how you can join the stream and ingest tables to return all tables that have been written into from a single workspace.

SQL

SELECT DISTINCT(stream.table_id)
FROM system.lakeflow.zerobus_stream AS stream INNER JOIN system.lakeflow.zerobus_ingest AS ingest ON (stream_id)
WHERE stream.workspace_id=:workspace_id