Skip to main content
Last updated on

Zerobus Ingest connector overview

Zerobus Ingest is a push-based ingestion API that writes data directly into Unity Catalog Delta tables. It is a serverless connector that automatically scales to handle incoming connections. It does not require configuring partitions or managing brokers.

With Zerobus Ingest, your "scaling strategy" is to open more connections. This streamlines ingestion workflows by eliminating the need for message bus infrastructure.

Any application that can integrate with Zerobus Ingest SDKs or communicate via supported APIs (gRPC and REST) can use Zerobus Ingest to push data efficiently into Delta tables.

The Zerobus Ingest buffers transmit data before adding it to a Delta table. This buffering creates an efficient and durable ingestion mechanism that supports a high volume of clients with variable throughput.

Once materialized in Delta format, the data becomes fully compatible with the comprehensive Databricks Data Intelligence Platform, enabling users to leverage familiar tools and capabilities for further data analysis and processing.

Concepts​

A data producer first opens a stream to Zerobus Ingest API and specifies a target Delta table, constructs a message matching its schema, and then pushes the message via the opened stream. The service makes the data durable, acknowledges the client's message, and materializes the data in the Delta table in an optimized fashion.

How Zerobus Ingest works

Streams​

A stream is a direct connection between your client and the Zerobus Ingest server. The SDKs use streams to facilitate long-lived, high-throughput connections.

  • Streams are only used in the gRPC API with the SDKs.
  • A stream ingests data to a single target table.
  • Event delivery ordering is guaranteed on the per-stream level.
  • Additional streams can be opened to write data to different tables.
  • Additional streams can be opened to increase client throughput.
    • Note: If your client round-robins messages to different streams, ordering will no longer be guaranteed.

Server​

The Zerobus Ingest service does not automatically create or manipulate tables. Users must create the table themselves. Tables and their schemas are the authoritative sources for the expectations of incoming data.

The Zerobus Ingest server accepts data sent to it by clients, validates that it fits the target table schema, and then writes it to the table. If the data fits in the table, it will be durable. An acknowledgment is then sent back to the client.

The service responsibilities include:

  • Schema validation of the message to the table.
  • Materializing the data in a timely manner in the target table.
  • Sending an acknowledgement to the client that the data is durable.

Client​

Client integration involves:

  • Selecting a target table.
  • Establishing a stream with the Zerobus Ingest service.
  • Constructing a schema-compatible message.
  • Sending the message.
  • Managing message acknowledgements.
  • Implementing recovery mechanisms in the case of client, stream, or server-side failures (e.g., connection issues, schema mismatches)

The Zerobus Ingest SDKs provide user-friendly methods to accomplish client integration, and the documentation offers examples of different development patterns. For custom integrations, the SDKs can serve as a reference for integration structure and recovery handling.

Get started with Zerobus Ingest​

note

If you have a client-side firewall, add the IP address used by Zerobus Ingest to your allowlist. To view IP addresses by region, see Databricks control plane addresses.

  1. Get a Zerobus Ingest URL.
  2. Create or identify the table you want to ingest data into.
  3. Create a service principal and grant privileges to the table.
  4. Write a client to start sending data.

For full instructions, see Use the Zerobus Ingest connector.

Durable Fallback Location​

If a breaking change is made to your target table after Zerobus Ingest makes your data durable before Zerobus Ingest has a chance to publish (push file to storage), the connector will make the data available in a separate folder within your table's storage location.

Data is persisted as Parquet files under the following path relative to the table root directory:

  • _zerobus/table_rejected_parquets/

This directory is created directly under the table’s physical root storage location, and all Parquet files generated by the service are written there.

This ensures that all persisted data remains within the table’s storage boundary and follows the same access controls and lifecycle policies as the table itself.

Cost​

Charges for Zerobus Ingest will be billed against the “Jobs Serverless” SKU. Pricing is available on the Lakeflow Connect pricing page.

Monitoring your usage​

There are two methods for monitoring your Zerobus Ingest connector usage.

Zerobus Ingest system tables​

You can monitor your usage via the Zerobus Ingest system tables.

Billing system table​

You can monitor your spend via the billable usage system table. Filter for Zerobus Ingest usage with:

  • billing_origin_product = 'LAKEFLOW_CONNECT'
  • product_features.lakeflow_connect.zerobus_request_type = 'GRPC' or 'HTTP'