Zerobus Ingest features
This page is an overview of what Zerobus Ingest in Lakeflow Connect offers. Each feature links to its full documentation.
- Always-on endpoint: serverless, on by default, scales automatically.
- Authentication methods: how clients authenticate, including service principal OAuth and external identity provider federation.
- Message blocking: ingestion methods and how to confirm durability.
- Acknowledgment callbacks: react to acknowledgments and errors asynchronously.
- Delta table features: partitioning, liquid clustering, and
VARIANTdata. - Rescue column: capture fields that don't fit the table schema instead of rejecting them.
- Durable fallback location: where data lands if it can't be published.
- Monitoring your usage: system tables for usage and spend.
Always-on endpoint
Zerobus Ingest is a serverless endpoint that is on by default for workspaces in supported regions. There's nothing to start, provision, or scale. After you have a target table and an authorized identity, you can begin pushing data, and capacity scales automatically. To get started, see Use Zerobus Ingest.
Authentication methods
Every Zerobus client authenticates with a short-lived Databricks OAuth token. When you use an SDK, it obtains and refreshes the token for you. On the other interfaces, such as the REST API, you obtain the token yourself and supply it with your requests. For setup steps and code, see Configure authentication.
OAuth with a service principal (default)
Authenticate with a Databricks service principal client ID and secret. This is the default method. It suits workloads that can hold a Databricks-managed secret.
External identity provider federation
Authenticate with a token from your own identity provider, such as Microsoft Entra ID, instead of a Databricks secret. The identity provider token is exchanged for a Zerobus-scoped Databricks token using RFC 8693 token exchange, so no long-lived Databricks secret is stored on the client.
Federation supports two identity models:
- Account-level federation: There is no Databricks service principal. Your identity from the identity provider is synced into the Databricks account through automatic identity management, and your client authenticates as that identity. This requires an account federation policy that registers your provider as a trusted issuer.
- Workload identity federation: Use a Databricks service principal that has a client ID and no secret, with a federation policy attached.
Microsoft Entra ID is the example on these pages, but the same federated flow works with other OpenID Connect (OIDC)-compatible identity providers.
Compare authentication methods
Method | Databricks service principal | Databricks client secret | When to use |
|---|---|---|---|
OAuth service principal | Yes | Yes | Your client can hold a Databricks-managed secret. |
Account-level federation | No | No | Your identity is governed only in your provider, with no Databricks service principal or secret. |
Workload identity federation | Yes, secretless | No | You want a Databricks service principal identity without storing a secret. |
Automatic token exchange in the SDK is available in the Python SDK 1.10.0 and above and the Rust SDK 2.11.0 and above. For setup steps and code, see Configure authentication.
Delta table features
Zerobus Ingest writes into standard Unity Catalog managed Delta tables, so your ingested data works with Delta table capabilities. For the requirements a target table must meet, see Table requirements.
- Partitioned tables: supported, subject to a partition-fan-out limit. Databricks recommends liquid clustering for high-cardinality keys. See Partitioned tables.
- Liquid clustered tables: the recommended layout, organizing data for query performance without the partition-fan-out limit. See Use liquid clustering for tables.
- Variant data: ingest semi-structured data into
VARIANTcolumns. See Supported data types.
Durable fallback location
If a breaking table change lands after Zerobus Ingest makes your data durable but before it can publish the data to the table, Zerobus Ingest writes that data as Parquet files to a fallback directory within your table's storage location, rather than dropping it. To find and reprocess this data after fixing the schema, see Recovering data from the durable fallback location.
Monitoring your usage
Monitor your Zerobus Ingest usage and spend through system tables. See Zerobus Ingest system table reference, and for spend, the billing system table under Cost.