Zerobus Ingest concepts
This page describes the core concepts of Zerobus Ingest in Lakeflow Connect: how the service works, its streams, server, and clients, and the data types it supports.
Jump to a concept:
- How Zerobus Ingest works: the end-to-end flow, from producer to queryable table.
- Server: what the Zerobus Ingest service is responsible for.
- Client: what your producer is responsible for.
- Streams: the connection your client opens to ingest data.
- How Zerobus Ingest scales: why it scales without capacity planning.
- Table requirements: what your target table and workspace must satisfy.
- Supported data types: the Delta-to-Protobuf type mapping.
How Zerobus Ingest works
A data producer first opens a stream to Zerobus Ingest API and specifies a target Delta table, constructs a message matching its schema, and then pushes the message through the opened stream. The service makes the data durable and acknowledges the client's message. It then materializes the data into the Delta table, in an optimized fashion, as a separate step. The acknowledgment confirms durability, not queryability. See Asynchronous communication for how this works and what it means for your client.
Zerobus Ingest is a serverless service that scales elastically with your workload. For how it scales, see How Zerobus Ingest scales below.
This section also covers the ways you connect to Zerobus Ingest and the shapes your data can take:
- API protocols: The API protocols, gRPC with SDKs, REST, and OpenTelemetry, and when to use each.
- Message types: The record formats, JSON, Protocol Buffers (protobuf), and Apache Arrow, and when to use each.
Server
The Zerobus Ingest service does not automatically create or manipulate tables. Users must create the table themselves. Tables and their schemas are the authoritative sources for the expectations of incoming data.
The Zerobus Ingest server accepts data sent to it by clients and validates that it fits the target table schema. If the record fits, the server makes it durable and acknowledges it to the client. Materializing the record into the Delta table, so that it becomes queryable, happens as a separate step shortly afterward.
The service responsibilities include:
- Schema validation of the message against the table.
- Making the record durable and acknowledging it to the client. The acknowledgment confirms durability, not that the record is queryable yet.
- Materializing the data into the target table in a timely manner, which is when it becomes queryable. For latency figures, see Latency.
Client
A client connects to Zerobus Ingest, sends records, and confirms they are durable. When you use a Zerobus Ingest SDK, the SDK handles most of this for you, so it helps to separate what you configure from what the SDK does automatically.
You configure or implement:
- Selecting a target table.
- Opening a stream to the Zerobus Ingest service.
- Constructing a schema-compatible message and sending it.
The SDK handles automatically:
- Message acknowledgments. The SDK runs the acknowledgment loop for you and surfaces durability confirmations through offsets or an acknowledgment callback. You only block on a specific record when your application needs to. See Asynchronous communication.
- Recovery. By default, the SDK reconnects and replays unacknowledged records on transient failures.
- You can turn the built-in recovery off and implement your own recovery mechanism instead. For what triggers recovery, the configuration options, and custom-recovery patterns, see Recovery and retry patterns.
You do not have to hand-write acknowledgment or recovery logic when you use an SDK. For custom integrations that don't use an SDK, the Zerobus SDK repository is a reference for integration structure and recovery handling.
Streams
A stream is a direct connection between your client and the Zerobus Ingest server, established over a persistent, bidirectional gRPC connection. The SDKs use streams to facilitate long-lived, high-throughput connections.
- Streams are only used in the gRPC API with the SDKs.
- A stream ingests data to a single target table.
- Open additional streams to write to different tables, or to scale a single client's throughput as high as your workload requires.
Streams are also the unit of ordering (see Ordering guarantees) and the unit by which Zerobus Ingest scales (see How Zerobus Ingest scales).
Ordering guarantees
Ordering is guaranteed per stream. Records are committed to the target table in the order they are enqueued on a single stream. There is no global ordering across streams. Several design points follow from this:
- If you spread records across multiple streams (for example, round-robin), there is no ordering guarantee across those streams.
- If your use case requires a single total order across many producers or streams, enforce that ordering in your application (for example, with a timestamp or sequence number you query on) rather than relying on ingestion order.
Why gRPC streaming
Because a stream's gRPC connection stays open, the client avoids the per-request setup cost of a stateless protocol and can push a continuous, high-volume flow of records down a single channel. This is what makes the SDKs the highest-throughput way to ingest. For the other interfaces (REST and OpenTelemetry) and when to choose each, see API protocols.
How Zerobus Ingest scales
Zerobus Ingest is designed for high scalability, and it reaches that scale without asking you to plan capacity. Two design choices make this possible:
- It's serverless. The service adds and removes capacity automatically as load changes, so you don't size brokers or provision partitions. You can open as many concurrent streams and write to as many tables as your workload needs.
- Streams are dynamic partitioning units. Rather than a fixed set of partitions that must be repartitioned and rebalanced to scale out, streams can be opened, closed, and rotated. Rotating streams lets the service rebalance capacity and resources as demand shifts, so you scale by opening more streams and running more producers while the service absorbs the rest.
The practical result is that a "hello world" client and a petabyte-scale workload run essentially the same code. The difference is how many producers and streams you run. This design has sustained ingestion of over 1 trillion records into a single Delta table. For the technical background, see the Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest blog post.
Table requirements
Zerobus Ingest writes to a Delta table you create and own. The target table and workspace must meet these requirements:
- Zerobus Ingest writes only to managed Delta tables. Writing to default storage is not supported.
- Zerobus Ingest does not write to storage secured through a private endpoint.
- Zerobus Ingest does not support recreating a target table.
- Table names support only ASCII letters, digits, and underscores.
- The workspace and the target table must both be in one of the supported regions.
For how records are validated against the table schema, see Schema management. For table features such as partitioning and liquid clustering, see Delta table features.
Supported data types
The following table shows the supported Delta types and their corresponding Protobuf types for ingestion.
Delta types | Protobuf types |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Decimal text, e.g. "123.45", "1e2", etc. |
|
|
|
|
|
|
|
Should be converted to |
|
Should be converted to |
|
Should be converted to |
|
|
|
|
The |
|
|
Over the gRPC SDKs and REST, ingest a Variant value as a JSON-encoded string with keys of type Supported formats include:
|
|