Message types
When you ingest with the Zerobus Ingest SDKs over gRPC, you choose how records are encoded on a given stream. Zerobus Ingest supports three message formats (JSON, Protocol Buffers (protobuf), and Apache Arrow), so you can trade off simplicity, type safety, and throughput for your workload. Every format is validated against your Delta table schema before the data is made durable. See Schema management.
Which format should you use?
Format | Best for | Notes |
|---|---|---|
JSON | Getting started and simple producers. | The simplest option, with no schema definition to compile. Convenient, but slower than the binary formats for high-volume workloads. |
Protocol Buffers | Production, row-oriented, high-volume streams. | Type-safe, compact binary encoding. Recommended for most production workloads. Requires a compiled schema. |
Apache Arrow | Columnar or batch-oriented workloads. | Sends Apache Arrow record batches directly, avoiding row-by-row serialization. Best when your data is already columnar or you ingest in batches. In Beta. See Use Arrow Flight with Zerobus Ingest. |
The snippets below show the shape of each format in the Python SDK. They assume you have already created the SDK client and know your target table. For the full setup (endpoint, table, service principal) and examples in every language, see Use Zerobus Ingest.
JSON
JSON is the simplest way to start: you send records as JSON objects with no schema definition to compile. It's ideal for getting up and running, prototyping, and producers where convenience matters more than raw throughput. For high-volume production workloads, a binary format (protobuf or Arrow) is more efficient.
Create a stream for JSON records by passing a table name to TableProperties with no descriptor:
table_properties = TableProperties(TABLE_NAME)
stream = sdk.create_stream(CLIENT_ID, CLIENT_SECRET, table_properties)
stream.ingest_record_offset({"device_name": "sensor-1", "temp": 22, "humidity": 55})
For the full JSON walkthrough, see Write a client.
Protocol Buffers
Protobuf provides a type-safe, compact binary encoding and is the recommended format for most production, row-oriented workloads. You define a protobuf schema that fits your target Delta table (see Protobuf schema), compile it, and the SDK ingests records record-by-record over gRPC.
Using protobuf takes three steps: generate a .proto schema that matches your table, compile it to a language module, then ingest records by passing the descriptor to TableProperties. The following example uses the Python SDK.
1. Generate a .proto schema from your table. The Python SDK includes a generate_proto tool that reads your Delta table and writes a matching schema:
python -m zerobus.tools.generate_proto \
--uc-endpoint "https://<workspace-id>.cloud.databricks.com" \
--client-id "<client-id>" \
--client-secret "<client-secret>" \
--table "main.default.air_quality" \
--output "record.proto" \
--proto-msg "AirQuality"
The generated schema uses proto2 syntax, with an optional field for each Delta column:
syntax = "proto2";
message AirQuality {
optional string device_name = 1;
optional int32 temp = 2;
optional int64 humidity = 3;
}
2. Compile the schema to a Python module with the protobuf compiler:
pip install "grpcio-tools>=1.60.0,<2.0"
python -m grpc_tools.protoc --python_out=. --proto_path=. record.proto
This generates record_pb2.py.
3. Ingest records by passing the compiled descriptor to TableProperties (the default for protobuf). The SDK uses the descriptor to serialize each record:
import record_pb2
descriptor_bytes = record_pb2.AirQuality.DESCRIPTOR.file.serialized_pb
table_properties = TableProperties(TABLE_NAME, descriptor_bytes)
stream = sdk.create_stream(CLIENT_ID, CLIENT_SECRET, table_properties)
record = record_pb2.AirQuality(device_name="sensor-1", temp=22, humidity=55)
stream.ingest_record_offset(record)
The example above uses the Python SDK. The tooling varies by language: some SDKs ship a generate_proto utility that generates a .proto from your table, while others (such as Go and TypeScript) compile an existing .proto. For the per-language steps, see the protobuf notes in each SDK tab of Write a client. For tool sources and complete examples, see the Zerobus SDK repository.
Apache Arrow
Apache Arrow ingestion is in Beta.
Apache Arrow ingestion sends Arrow RecordBatch data directly over the same gRPC connection, instead of converting each row to JSON or protobuf first. It's the best choice when your application already produces Arrow data or when you ingest rows in batches, especially for wide, numeric-heavy, or analytics-oriented schemas where row-by-row serialization adds overhead.
Arrow is also a good fit for very large batches. Unlike the JSON and protobuf batch methods, which are all-or-nothing and bounded by the per-message size limit, the Arrow Flight path splits a large batch into smaller transport messages that are sent and acknowledged individually. See Arrow Flight batches are the exception and Use Arrow Flight with Zerobus Ingest.
Open an Arrow stream with a pyarrow.Schema and ingest RecordBatch data:
stream = sdk.create_arrow_stream(TABLE_NAME, schema, CLIENT_ID, CLIENT_SECRET)
stream.ingest_batch(batch)
For the full Arrow Flight walkthrough, including schema definition, batching, and compression, see Use Arrow Flight with Zerobus Ingest.
Related
- API protocols: The API protocols these formats travel over.
- Schema management: How records are validated against your table.
- Supported data types: Supported Delta and Protobuf data types.