Stream changes to Zerobus Ingest with Debezium Server
The Databricks Zerobus Ingest sink is part of Debezium Server. For the sink options, see Debezium's Databricks Zerobus Ingest sink reference.
Maintained by: the Debezium open-source project.
Debezium Server is the standalone runtime for Debezium, the open-source change data capture (CDC) project. It reads a database's change log and turns every insert, update, and delete into a change event. Its native Databricks Zerobus Ingest sink writes those events into Unity Catalog Delta tables through Zerobus Ingest. One Debezium Server process sits between the database and Delta, so you don't need to run a message bus or a Kafka Connect cluster for CDC into the lakehouse.
The sink takes change events from Debezium's log-based source connectors, such as PostgreSQL, MySQL, SQL Server, and Oracle. This page uses PostgreSQL. For another database, only the debezium.source.* properties and, in typed mode, the routing pattern change. MySQL, SQL Server, and Oracle also emit schema-change events, to a destination named after topic.prefix, so set debezium.source.include.schema.changes=false for those connectors. Otherwise their schema-change events go to that destination, which isn't a table in your catalog.
Choose a payload mode
The sink writes in one of two payload modes. Pick one before you create tables, because each mode needs a different table schema.
Consideration | Envelope mode | Typed mode (default) |
|---|---|---|
What lands | The full change event in a fixed nine-column table: operation, before and after row images, source position, and an | One row per change, in a table with the source table's own columns |
Source schema changes | Land inside the | Debezium Server stops until you add the new column to the Delta table. |
Updates and deletes | Kept as events. Build current-state or history tables downstream. | Each update is a new row. A delete is a new row with |
Duplicates | Deduplicate on | Rows have no event ID. |
Choose it when | You want history, many source tables in one place, or a pipeline that keeps running through source schema changes | You want queryable rows with the source's columns right away, and the source schema rarely changes |
In typed mode, the Delta table's schema must match every change event. When a source table gains a column, Zerobus Ingest rejects the event with error 4044, and Debezium Server stops for every table it captures until you add the column to the Delta table. See Change a source table's schema.
When to use Debezium Server
- You want row-level changes from a database, such as PostgreSQL, MySQL, SQL Server, or Oracle, in Delta tables.
- You already use Debezium connectors and want their change events in Unity Catalog directly.
- You want every change kept, including deletes, so that you can rebuild history or apply the changes downstream.
How it works
- Interface: With
debezium.sink.type=zerobus, the sink uses the Zerobus Ingest Java SDK over gRPC and opens one stream per target table. Debezium also offers a REST route (zerobus-rest, one HTTP request per record) and a route through the Kafka-compatible API. This page covers the gRPC route, which Debezium recommends. - Envelope mode: The sink writes each change event unchanged. Set
debezium.sink.zerobus.tableto send every captured table to one envelope table. Thedestinationandtarget_tablecolumns record where each event came from. Don't apply theExtractNewRecordStatetransform in this mode, because it removes the operation type. - Typed mode: The
ExtractNewRecordStatetransform flattens each change event to the row's new values, and aRegexRoutertransform maps each source table to its own Delta table. The sink writes each field to the table column with the same name, using the target column's type. For the full mapping, see the Debezium data type mapping. - Authentication: The sink authenticates with a service principal's OAuth client ID and secret.
- Delivery: The sink commits source offsets only after Zerobus Ingest acknowledges the records for every target table in a batch. Each event can be delivered one or more times, so a replayed batch can produce duplicate rows. See Handle duplicates.
Requirements
- Debezium Server 3.7 and above, the release line that includes the Zerobus Ingest sink. See the Debezium 3.7 release notes and Debezium Server installation.
- A Java 21 runtime for Debezium Server.
- A source database set up for CDC, as the Debezium connector documentation describes for your database. For example, PostgreSQL needs logical decoding and a publication.
- A workspace in a region that supports Zerobus Ingest. See Availability.
- A service principal with an OAuth secret. See Configure authentication. Note the service principal's application ID and client secret.
- Target tables that meet the Zerobus Ingest table requirements.
- Change Data Feed and column mapping turned off on the target tables, as the Debezium sink requires.
- Network access from the Debezium Server host to your Zerobus Ingest endpoint and your workspace URL on port 443.
Step 1: Create the target table and grant access
Zerobus Ingest doesn't create tables, so create them before you start Debezium Server.
- Envelope mode
- Typed mode
Create one table with the envelope's nine columns. The key, value, source_position, and headers columns are VARIANT, so you can query any field of the change event.
CREATE TABLE <catalog>.<schema>.customers_events (
target_table STRING,
destination STRING,
partition INT,
operation STRING,
idempotency_key STRING,
key VARIANT,
value VARIANT,
source_position VARIANT,
headers VARIANT
) USING DELTA;
Create one table per source table, with one column per source column and a __deleted column. Declare __deleted as STRING, not BOOLEAN, because Debezium writes the string "true" or "false". The following table matches a source table customers (id, name, balance, tier):
CREATE TABLE <catalog>.<schema>.customers (
id INT,
name STRING,
balance DECIMAL(12,2),
tier STRING,
__deleted STRING
) USING DELTA;
Grant the service principal USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT and MODIFY on each target table, as Configure authentication describes.
Step 2: Get your endpoints
The sink needs two values:
debezium.sink.zerobus.endpoint: your Zerobus Ingest endpoint, without thehttps://prefix.debezium.sink.zerobus.workspace.url: your workspace URL, with thehttps://prefix. The sink uses it to get OAuth tokens.
endpoint:<workspace-id>.zerobus.<region>.cloud.databricks.comworkspace.url:https://<workspace-url>
To find your workspace ID, region, and URL, see Get your workspace URL and Zerobus Ingest endpoint.
Step 3: Configure Debezium Server
Debezium Server reads config/application.properties in its installation directory. The file has a source block, which depends on your database, and a format and sink block, which depends on the payload mode.
Debezium Server doesn't run Kafka. Connectors that track schema changes, such as MySQL, SQL Server, and Oracle, also keep a schema history. Debezium's default store for it is Kafka, so those connectors need a store outside Kafka. Debezium names FileSchemaHistory for non-Kafka deployments. To use it, set debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory and point debezium.source.schema.history.internal.file.filename at a file on durable storage, such as data/schema-history.dat. PostgreSQL doesn't need a schema history.
The following block captures public.customers from a PostgreSQL database named inventory. For another database, replace it with the properties that the Debezium documentation for that connector describes. Keep the offset settings. If two sources write to the same target table, give each one its own topic.prefix. For PostgreSQL, each connector also needs its own debezium.source.slot.name and debezium.source.publication.name. The prefix is part of each event's destination and source position, so two sources with the same prefix can produce the same idempotency_key. In typed mode, change the route pattern to match the new prefix.
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.database.hostname=<database-host>
debezium.source.database.port=5432
debezium.source.database.user=<database-user>
debezium.source.database.password=${PG_PASSWORD}
debezium.source.database.dbname=inventory
debezium.source.topic.prefix=cdc
debezium.source.table.include.list=public.customers
debezium.source.plugin.name=pgoutput
debezium.source.publication.name=dbz_publication
debezium.source.publication.autocreate.mode=disabled
debezium.source.slot.name=dbz_zerobus
debezium.source.snapshot.mode=initial
# Keep offsets on durable storage so that a restart resumes from the last
# committed position instead of taking a new snapshot.
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=1000
# Send DECIMAL values as exact text.
debezium.source.decimal.handling.mode=string
Before you start Debezium Server, create the publication that the block names. The block sets publication.autocreate.mode=disabled, so Debezium doesn't create the publication, and it stops at startup if dbz_publication doesn't exist. Debezium also doesn't add tables to an existing publication, so the publication must include the captured table:
CREATE PUBLICATION dbz_publication FOR TABLE public.customers;
If the publication already exists, add public.customers to it with ALTER PUBLICATION. To use a different publication, set debezium.source.publication.name to its name.
Set the captured table's replica identity to FULL as well. With the default identity, an update event's before holds only the primary key columns, and a delete event's before holds only the key. An unchanged TOASTed column (a large value that PostgreSQL stores out of line) is left out of an update event, and Debezium sends its unavailable.value.placeholder value in its place. With FULL, each update and delete event carries the previous values of all columns, and TOAST values are included:
ALTER TABLE public.customers REPLICA IDENTITY FULL;
Run it before you start Debezium Server, with the publication statement above.
Add the format and sink block for your payload mode. Debezium Server resolves ${...} expressions from environment variables, so the client secret stays out of the file.
- Envelope mode
- Typed mode
debezium.format.key=json
debezium.format.key.schemas.enable=false
debezium.format.value=json
debezium.format.value.schemas.enable=false
debezium.sink.type=zerobus
debezium.sink.zerobus.endpoint=${ZEROBUS_ENDPOINT}
debezium.sink.zerobus.workspace.url=${DATABRICKS_WORKSPACE_URL}
debezium.sink.zerobus.client.id=${DATABRICKS_CLIENT_ID}
debezium.sink.zerobus.client.secret=${DATABRICKS_CLIENT_SECRET}
debezium.sink.zerobus.table=<catalog>.<schema>.customers_events
debezium.sink.zerobus.payload.mode=envelope
debezium.sink.zerobus.record.format=json
debezium.sink.zerobus.json.flexible.fields.encoding=string
debezium.sink.zerobus.idempotency.mode=source
debezium.sink.zerobus.tombstone.handling.mode=event
debezium.sink.zerobus.metrics.log.interval=10
Don't apply the ExtractNewRecordState transform. The sink needs the whole change event. Routing transforms such as RegexRouter are fine.
debezium.transforms=unwrap,route
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.transforms.unwrap.delete.tombstone.handling.mode=rewrite
debezium.transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
debezium.transforms.route.regex=cdc\\.public\\.(.*)
debezium.transforms.route.replacement=<catalog>.<schema>.$1
debezium.format.key=json
debezium.format.key.schemas.enable=false
debezium.format.value=json
debezium.format.value.schemas.enable=false
debezium.sink.type=zerobus
debezium.sink.zerobus.endpoint=${ZEROBUS_ENDPOINT}
debezium.sink.zerobus.workspace.url=${DATABRICKS_WORKSPACE_URL}
debezium.sink.zerobus.client.id=${DATABRICKS_CLIENT_ID}
debezium.sink.zerobus.client.secret=${DATABRICKS_CLIENT_SECRET}
debezium.sink.zerobus.payload.mode=typed
debezium.sink.zerobus.record.format=json
debezium.sink.zerobus.metrics.log.interval=10
The route transform turns each destination name, such as cdc.public.customers, into <catalog>.<schema>.customers. The destination name's format depends on the connector. For example, the SQL Server connector with database.names adds the database name, as in cdc.inventory.dbo.orders, so its pattern is cdc\\.[^.]+\\.dbo\\.(.*). The unwrap transform also replaces a NULL with the source-defined default when the column has one, because replace.null.with.default defaults to true. To keep explicit NULLs, add debezium.transforms.unwrap.replace.null.with.default=false to the block above.
Sink options
The following options are the ones you're most likely to set. For the full list, including the envelope filters and Protobuf records, see the Debezium Server Databricks Zerobus Ingest reference.
Option | Description |
|---|---|
| Required. Your Zerobus Ingest endpoint, without |
| Required. Your workspace URL, with |
| Required. The service principal's application ID and OAuth secret. |
|
|
| Optional. Writes every event to this one |
| Envelope mode only. |
| Envelope mode only. |
| Envelope mode only. |
| The maximum number of unacknowledged records per stream. The default is 50,000. |
| The maximum number of streams, and so tables, open at one time. The default is 100. |
| Writes a metrics summary to the log every N batches. The default, 0, turns it off. |
Step 4: Start Debezium Server and verify
Download and unpack the Debezium Server distribution, as Debezium Server installation describes. Put the file from Step 3 at debezium-server/config/application.properties. Then set the values that the configuration reads from the environment, create the data directory for the offsets file, and start the server. Debezium Server doesn't create that directory, and without it the offsets aren't saved, so a restart takes a new snapshot. In production, put data on durable, writable storage.
export PG_PASSWORD=<database-password>
export ZEROBUS_ENDPOINT=<zerobus-endpoint> # your endpoint from Step 2
export DATABRICKS_WORKSPACE_URL=https://<workspace-url>
export DATABRICKS_CLIENT_ID=<service-principal-uuid>
export DATABRICKS_CLIENT_SECRET=<client-secret>
cd debezium-server
mkdir -p data
./run.sh
Each target table gets its own stream. Look for these log messages:
io.debezium.server.databricks.zerobus.ZerobusChangeConsumer: Zerobus gRPC sink connected: endpoint=<zerobus-endpoint>
io.debezium.pipeline.ChangeEventSourceCoordinator: Snapshot ended with SnapshotResult [status=COMPLETED
io.debezium.server.databricks.zerobus.ZerobusChangeConsumer: Opening Zerobus json stream for table '<catalog>.<schema>.customers_events'
com.databricks.zerobus.BaseZerobusStream: All records have been flushed
Then query the table. The initial snapshot arrives first, followed by every change after it. In envelope mode, snapshot rows have operation set to read.
- Envelope mode
- Typed mode
SELECT
operation,
variant_get(value, '$.before.id', 'INT') AS before_id,
variant_get(value, '$.after.id', 'INT') AS after_id,
variant_get(value, '$.after.balance', 'STRING') AS balance,
variant_get(value, '$.source.lsn', 'BIGINT') AS source_lsn,
idempotency_key
FROM <catalog>.<schema>.customers_events
ORDER BY source_lsn NULLS LAST;
An update has both before and after images. A delete has only before, and its tombstone row has a NULL value, so it sorts last here. The $.source.lsn path is specific to PostgreSQL. Every event also has a source_position column with the source partition and offset.
To build current-state or history tables from the envelope, see Additional resources.
SELECT id, name, balance, tier, __deleted
FROM <catalog>.<schema>.customers
ORDER BY id;
An insert and a later update to the same source row appear as two rows, one for each version. A delete appears as a row with __deleted set to "true".
Change a source table's schema
In envelope mode, a new source column appears inside the value column of the next event, for example as $.after.<column>. The Delta table doesn't change, and the pipeline keeps running.
In typed mode, your Delta table's schema is the contract for what Zerobus Ingest accepts, and Zerobus Ingest doesn't change it for you. Add the column to the Delta table before you add it to the source table:
ALTER TABLE <catalog>.<schema>.customers ADD COLUMN region STRING;
Then restart Debezium Server before you change the source table. The restart opens new streams against the updated table, and it resumes from the stored offsets without taking a new snapshot.
If a change event carries a column that the Delta table doesn't have, Zerobus Ingest rejects it with error 4044 and the sink fails the whole batch. Debezium Server then stops for every table it captures, not only the changed one, and it doesn't commit the batch's offsets. After you add the missing column and restart Debezium Server, the held events land. See Schema management.
Handle duplicates
Each event can be delivered one or more times. If Debezium Server stops before it commits a batch's offsets, such as after a typed-mode schema error, it sends that batch again when it restarts. A clean restart resumes from the stored offsets without taking a new snapshot. See Delivery guarantees.
- Envelope mode: A replayed event has the same
idempotency_key. Count unique changes withcount(DISTINCT idempotency_key), and remove duplicates before you apply changes downstream. The identity includes the record key, which isnullfor a table without a primary key, so rows from that table's initial snapshot can share an identity. Don't deduplicate such a table onidempotency_keyalone. - Typed mode: Rows have no event ID, so a replayed event can't be told apart from a later change that repeats the same values, such as a row that changes from A to B and back to A. Use envelope mode, with its
idempotency_key, when you need exact deduplication.
Troubleshooting
Symptom | Cause | Fix |
|---|---|---|
| Typed mode: a source column has no matching column in the Delta table, usually after a source schema change | Add the column to the Delta table with |
Debezium Server stops for every table after one table's schema changed | The sink fails the whole batch, and the batch spans tables | Fix the table named in the error. The other tables resume with it. |
Every envelope row has | An | Remove the |
Duplicate rows after a restart | A batch replayed because its offsets weren't committed | Deduplicate downstream. See Handle duplicates. |
Additional resources
- BrickTalk: Zerobus Ingest & CDC Workflows: the recorded session with the team behind the sink.
- Push-based CDC into Unity Catalog with Zerobus Ingest and Debezium: the Databricks blog post on the architecture.
- How to stream PostgreSQL CDC into Databricks with Debezium: a Community technical blog walkthrough in envelope mode, including current-state and history tables with AUTO CDC.
- Debezium Zerobus example: a runnable SQL Server example in typed mode.
- Schema management: plan table schema changes.
- Debezium Server Databricks Zerobus Ingest reference: every sink option, the envelope row contract, and the data type mapping.