Skip to main content
Last updated on

Zerobus Ingest connector limitations

This page lists limitations when using the Zerobus Ingest connector in Lakeflow Connect.

Availability​

The Zerobus Ingest connector is available in the following regions:

  • us-east-1
  • us-east-2
  • us-west-2
  • ca-central-1
  • ap-southeast-1
  • ap-southeast-2
  • ap-northeast-1
  • ap-northeast-2
  • ap-south-1
  • eu-central-1
  • eu-west-1
  • eu-west-2
  • sa-east-1

All of the available regions provide multi-zonal service availability.

Latency​

Latency reflects both the durability acknowledgment and the time to materialize records into the target Delta table. Actual times vary based on region alignment and workload characteristics.

  • Time to durability
    • P95 ≤ 500 ms
    • P50 ≤ 200 ms
  • Time to table
    • P95 ≤ 30 sec
    • P50 ≤ 5 sec

Throughput restrictions​

  • 100 MB/second per stream (benchmarked with 1KB-sized messages)
  • 10 GB/second per target table
  • 15,000 records per second per stream

To achieve maximum throughput, a client app and endpoint should be in the same geographic region. If you need higher throughput, contact your Databricks account representative.

Delivery guarantees​

The Zerobus Ingest connector only provides at-least-once guarantees.

Quotas​

Below are the default quotas for the Zerobus Ingest connector. If you need higher performance, contact your Databricks account representative.

gRPC

  • 100MB per second throughput per stream
  • 10GB per second throughput per target table

REST

  • 10,000 requests per second

Partitioned tables​

When writing to partitioned tables, the Zerobus Ingest connector does not support writing to more than 1000 partitions within 5 second intervals.

Catalog-managed commits​

Zerobus Ingest does not support catalog-managed commits. Do not use Zerobus Ingest for Delta tables with catalog-managed commits enabled.

Workspace and Target table​

The following workspace and target table conditions are required for ingestion.

  • The connector supports writing only to managed Delta tables. Writing to default storage is not supported.
  • The connector does not support writing to liquid clustered tables.
  • The connector does not support recreating a target table.
  • The connector only supports table names with ASCII letters, digits, and underscores.
  • Both the workspace and the target table need to be in one of the available regions, and both in the same region.

Schema evolution​

Zerobus Ingest will never auto-evolve your target table.

Zerobus Ingest supports continuous ingestion when nullable Delta columns are added to the target table. Missing columns are filled with NULL values, allowing you to send records with missing fields.

Protobuf schema​

The protobuf schema definition must match 1:1 with the Delta table schema (excluding extra nullable delta columns, which are considered a non-breaking schema change). If the schema does not match, the API returns an error. This includes:

  • Different number of columns

  • Different column names

  • Different column optionality (nullable and non-nullable)

  • The connector does not support proto schemas with more than 2000 columns.

  • The connector only supports table and column names with ASCII letters, digits, and underscores.

  • The connector does not support using a different proto schema for “stream creation” and “ingest record” operations.

Record size​

Each message is limited to 10 MB. Record size max is 10,485,760 bytes. Headers required for communication take up 19 bytes.

Type support​

The following table shows the supported Delta types and their corresponding Protobuf types for ingestion.

Delta types

Protobuf types

INTEGER

int32

STRING

string

FLOAT

float

LONG

int64

SHORT

int32

DOUBLE

double

BOOLEAN

bool

BINARY

bytes

DATE

Should be converted to int32 (number of days since epoch).

int32

TIMESTAMP

Should be converted to int64 (epoch time in microseconds).

int64

ARRAY<TYPE>

repeated TYPE

MAP<K,V>

map<K,V>

The map Protobuf syntactic sugar is only available for Protobuf compilers version 3 and above.

STRUCT<FIELDS>

message Nested { FIELDS }

VARIANT

Variant must be ingested as a JSON-encoded string with keys of type STRING. The connector writes the data unshredded into the column.

Supported formats include:

  • Objects: "{\"id\":0,\"example\":\"this is variant example\"}"
  • Primitives: "5", "3.14", "\"string\""
  • Arrays: "[1,2,3]"

string