Asynchronous communication

Communication on a stream is asynchronous and bidirectional. Your client pushes records continuously without waiting for each one to be confirmed, and the server sends acknowledgments back over the same connection as records become durable. This decoupling is what lets a single client sustain high throughput: it keeps pushing while acknowledgments arrive in the background.

Offsets and the acknowledgment loop

Every submission on a stream, whether a single record or a batch, is assigned a logical offset that marks its position in that stream. Rather than acknowledging each submission individually, the server reports cumulative durability progress through the highest committed offset it has made durable so far. Because offsets are ordered, one acknowledgment confirms that submission and all earlier ones.

This is the acknowledgment loop, and it's what keeps the connection both fast and reliable:

1. The client pushes records and holds them in a local in-flight buffer.
2. The server persists records durably and periodically sends back the highest committed offset.
3. On receiving that offset, the client safely purges every buffered record up to it, because those records are now durable.

When you use a Zerobus Ingest SDK, the SDK runs this loop for you. It tracks offsets, maintains the in-flight buffer, and processes acknowledgments in the background while your producer keeps pushing. You don't implement the loop yourself. What you optionally control is how you observe durability:

• Keep sending; the SDK processes acknowledgments as they arrive.
• Block on an offset only when your application must wait for a specific record to be durable. See below.
• Register an acknowledgment callback to react to confirmations and errors asynchronously, without blocking. See Acknowledgment callbacks.

You would only implement the offset-tracking and buffering loop yourself if you build a custom client that does not use an SDK.

The in-flight buffer is bounded by a configurable in-flight record limit. Ingestion is asynchronous until the buffer fills; at that point, ingest calls block until acknowledgments arrive and free up space. Tune the limit for your workload, and note that buffered records consume client memory while they're in flight. For the option and its default, see the Zerobus SDK repository.

If the connection is interrupted, records still in the in-flight buffer (those beyond the last committed offset) have not been confirmed durable, so they can be replayed. See Recovery and retry patterns.

The acknowledgment confirms durability, not queryability. A committed offset means those records are durably persisted and will not be lost. Zerobus Ingest materializes durable records into the Delta table as a separate step shortly afterward, at which point the data becomes queryable in approximately 5 seconds. For more on latency, see Latency.

Waiting on a record versus maximizing throughput

You wait on a record's offset when your application must block further execution until that specific record is known to be durable, for example before it acknowledges the work to an upstream system. Waiting is about application-level synchronization, not a requirement for durability. A record becomes durable through the acknowledgment loop whether or not you block on it.

Blocking has a throughput cost:

• Waiting after every record turns ingestion into an effectively synchronous workflow. Blocking on each message before sending the next prevents the client from reaching Zerobus Ingest's full throughput.
• High-throughput ingestion is continuous and asynchronous. The client keeps sending records while acknowledgments arrive for groups of prior records, rather than pausing on each one. Wait on a specific offset only at the checkpoints where your application genuinely needs that guarantee, or use an acknowledgment callback to track progress without blocking.

For the ingestion methods, when to block on an offset, and how acknowledgment callbacks work, see Message blocking and acknowledgment.

Ordering on a stream

Acknowledgments and offsets are per stream: ordering is guaranteed within a single stream, not globally across streams. For how per-stream ordering works and how to design around it, see Ordering guarantees.