Zerobus Ingest features
This page is an overview of what Zerobus Ingest in Lakeflow Connect offers. Each feature links to its full documentation.
- Always-on endpoint: serverless, on by default, scales automatically.
- Message blocking: ingestion methods and how to confirm durability.
- Acknowledgment callbacks: react to acknowledgments and errors asynchronously.
- Delta table features: partitioning, liquid clustering, and
VARIANTdata. - Rescue column: capture fields that don't fit the table schema instead of rejecting them.
- Durable fallback location: where data lands if it can't be published.
- Monitoring your usage: system tables for usage and spend.
Always-on endpoint
Zerobus Ingest is a serverless endpoint that is on by default for workspaces in supported regions. There's nothing to start, provision, or scale. Once you have a target table and a service principal with access to it, you can begin pushing data, and capacity scales automatically as you open more streams. To get started, see Use Zerobus Ingest.
Delta table features
Zerobus Ingest writes into standard Unity Catalog managed Delta tables, so your ingested data works with Delta table capabilities. For the requirements a target table must meet, see Table requirements.
- Partitioned tables: supported, subject to a partition-fan-out limit. Databricks recommends liquid clustering for high-cardinality keys. See Partitioned tables.
- Liquid clustered tables (Beta): the recommended layout, organizing data for query performance without the partition-fan-out limit. See Use liquid clustering for tables.
- Variant data: ingest semi-structured data into
VARIANTcolumns. See Supported data types.
Durable fallback location
If a breaking table change lands after Zerobus Ingest makes your data durable but before it can publish the data to the table, Zerobus Ingest writes that data as Parquet files to a fallback directory within your table's storage location, rather than dropping it. To find and reprocess this data after fixing the schema, see Recovering data from the durable fallback location.
Monitoring your usage
Monitor your Zerobus Ingest usage and spend through system tables. See Zerobus Ingest system table reference, and for spend, the billing system table under Cost.