API protocols
Zerobus Ingest exposes one endpoint with several API protocols: gRPC (through the SDKs), REST, OpenTelemetry (OTLP), and Kafka-compatible APIs. All of them write directly into Unity Catalog Delta tables, so you choose the protocol that best fits each producer.
Which protocol should you use?
Protocol | Best for | Why |
|---|---|---|
gRPC (SDKs) | High-volume streaming producers: change data capture, clickstream, log and event forwarders. | Persistent connections deliver the highest sustained throughput and preserve per-stream ordering. Available in Python, Java, Rust, Go, TypeScript, and (in Beta) C++ and C# / .NET. See Write a client. |
REST | Large fleets of lightweight or "chatty" clients, such as edge and IoT devices that report infrequently. | Stateless: each request stands alone, so clients don't hold an open connection. Simpler for devices that send data occasionally. See the REST example. |
OpenTelemetry (OTLP) | Observability pipelines already emitting traces, logs, and metrics. | Point your existing OpenTelemetry SDKs or collectors at the endpoint with no custom integration. See Ingest OpenTelemetry data with Zerobus Ingest. |
Kafka-compatible APIs (Beta) | Producers that already speak the Kafka protocol, or tooling that emits to Kafka, when you want that data in Delta with minimal code change. | Reuse an existing Kafka producer without a Databricks SDK. Point it at the endpoint and produce to a topic named after your target table. See Use Kafka-compatible APIs with Zerobus Ingest. |
gRPC with the SDKs
The Zerobus SDKs wrap a persistent, bidirectional gRPC connection called a stream. Because the connection stays open, gRPC achieves the highest sustained throughput and is the recommended path for continuous, high-volume ingestion. Each open stream is a long-lived connection, so a client's throughput scales with the number of streams it opens.
The SDKs handle connection management, offset tracking, and recovery for you, and are available in Python, Java, Rust, Go, TypeScript, and (in Beta) C++ and C# / .NET. They provide equivalent behavior across languages, so choose the one that fits your application. Over gRPC, the SDKs support JSON, protobuf, and Apache Arrow record formats. See Message types.
To write a client with the SDKs, including a per-language example for each, see Write a client.
REST
The REST interface is stateless: each request completes on its own without holding an open connection. This makes it a strong fit for large fleets of lightweight or intermittent producers, for example edge and IoT devices that report status infrequently, where maintaining a persistent connection per device would be impractical.
REST ingestion is governed by a request-rate quota (see Zerobus Ingest quotas). For a high-volume producer, the SDKs over gRPC will sustain higher throughput than issuing many individual REST requests. Reserve REST for producers that send infrequently or can't hold a persistent connection.
OpenTelemetry (OTLP)
Zerobus Ingest includes a native OpenTelemetry Protocol endpoint. If your systems already produce OpenTelemetry traces, logs, and metrics, you can point your existing OTLP exporters or collectors at Zerobus Ingest and land that telemetry directly in Delta tables you own, usually with just a configuration change. See Ingest OpenTelemetry data with Zerobus Ingest.
Kafka-compatible APIs
The Kafka-compatible APIs are in Beta.
Zerobus Ingest offers Kafka-compatible producer APIs, so you can ingest with any Apache Kafka producer client without a Databricks SDK. You point an existing Kafka producer at the endpoint and produce to a topic named after your target table, and the records land in a Unity Catalog Delta table. This is the best fit when you already run a Kafka producer, or a collector or agent that emits to Kafka, and want to route that data into Delta with minimal code change.
The Kafka-compatible APIs implement the producer-side subset of the Kafka protocol and are write-only: consumer, admin, and transactional APIs are not available. They accept JSON records only. Throughput is governed by a per-workspace request-rate quota rather than by individual connections. For the specific quota, see Zerobus Ingest quotas. For the highest throughput, per-record acknowledgments, and automatic recovery, use a Zerobus SDK over gRPC instead.
For setup, authentication, and a producer example, see Use Kafka-compatible APIs with Zerobus Ingest.
Related
- Message types: The record formats available over these protocols.
- Use Zerobus Ingest: Write a client using the SDKs or REST API.
- Ingest OpenTelemetry data with Zerobus Ingest: Ingest OpenTelemetry data.
- Zerobus Ingest release stages: Release stages for each interface and record format.
- Use Kafka-compatible APIs with Zerobus Ingest: Use Kafka-compatible producer APIs.