Zerobus Ingest connector quotas
This page describes the default quotas for the Zerobus Ingest connector in Lakeflow Connect, along with how it works with your Delta tables, schema, and data types.
Availability
The Zerobus Ingest connector is only available in some regions. For a list of supported regions, see Ingestion availability.
Latency
Latency reflects both the durability acknowledgment and the time to materialize records into the target Delta table. Actual times vary based on region alignment and workload characteristics.
- Time to durability
- P50 ≤ 150 ms
- Time to table
- P50 ≤ 5 sec
Quotas
The following table lists the default quotas for the Zerobus Ingest connector.
Item | Default quota | Need more? |
|---|---|---|
Throughput per stream (gRPC) | 100 MB/second (benchmarked with 1 KB messages) | Adjustable |
Throughput per target table (gRPC) | 10 GB/second | Adjustable |
Records per second per stream | 100,000 (benchmarked with 1 KB messages) | Adjustable |
REST requests per second | 10,000 | Adjustable |
These are the default throughputs the connector is provisioned with, and they scale to meet higher workloads. To raise any of them, contact your Databricks account representative. For maximum throughput, keep your client app and the endpoint in the same geographic region.
Delivery guarantees
The Zerobus Ingest connector provides at-least-once guarantees.
Buffered data retention
If the Zerobus Ingest service cannot flush buffered data to your Delta table, it holds the data in service-managed durable storage and occasionally emits per-stream warning events.
If the data remains unflushed for 28 days, the service deletes the buffered data and emits a per-stream discard event recording the number of records and bytes dropped. You cannot recover the data after deletion.
Table behavior
How Zerobus Ingest works with your Delta tables.
Partitioned tables
When writing to partitioned tables, Zerobus Ingest does not support writing to more than 100 partitions in any 5-second interval. For best throughput, minimize the number of partitions in each 5-second interval.
Databricks recommends using liquid clustered tables with Zerobus Ingest instead of partitioning.
Workspace and target table
Ingestion works with the following workspace and target table setup.
- The connector supports writing to managed Delta tables.
- The connector does not support recreating a target table.
- The connector only supports table names with ASCII letters, digits, and underscores.
- Both the workspace and the target table need to be in one of the available regions.
Liquid clustered tables
When using the Zerobus Ingest connector with liquid clustered tables, it's recommended to keep predictive optimization enabled for the target table. The connector writes data into the table, but optimal data clustering is applied asynchronously by the predictive optimization service. Disabling predictive optimization can result in suboptimal query performance on the ingested data.
Schema and data types
The following sections explain how to define your schema and the data types Zerobus Ingest supports.
Schema evolution
Zerobus Ingest will never auto-evolve your target table.
Zerobus Ingest supports continuous ingestion when nullable Delta columns are added to the target table. Missing columns are filled with NULL values, allowing you to send records with missing fields.
Protobuf schema
The protobuf schema definition must match 1:1 with the Delta table schema (excluding extra nullable delta columns, which are considered a non-breaking schema change). If the schema does not match, the API returns an error. This includes:
-
Different number of columns
-
Different column names
-
Different column optionality (nullable and non-nullable)
-
The connector does not support proto schemas with more than 2000 columns.
-
The connector only supports table and column names with ASCII letters, digits, and underscores.
-
The connector does not support using a different proto schema for "stream creation" and "ingest record" operations.
Record size
Each message has a maximum size of 10 MB. The record size maximum is 10,485,760 bytes. Headers required for communication take up 19 bytes.
Type support
The following table shows the supported Delta types and their corresponding Protobuf types for ingestion.
Delta types | Protobuf types |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Decimal text, e.g. "123.45", "1e2", etc. |
|
|
|
|
|
|
|
Should be converted to |
|
Should be converted to |
|
Should be converted to |
|
|
|
|
The |
|
|
Variant must be ingested as a JSON-encoded string with keys of type Supported formats include:
|
|