Send data to Zerobus Ingest from dlt
dlt's Databricks destination has a Zerobus mode. Set insert_api to "zerobus", and append loads go through Zerobus Ingest. That mode also uses a SQL connection to create the table and to hold dlt's state tables. See dlt's Databricks destination. This page shows a different route: a destination that writes only through the Zerobus Ingest Python SDK, with no SQL warehouse connection at load time. The SDK gets and refreshes the OAuth token itself.
Maintained by: the dltHub open-source project, in the dlt-hub/dlt repository. The destination on this page is your own code, not part of dlt.
This page shows how to load rows from a dlt pipeline into a Zerobus Ingest table. dlt loads data from sources into a destination. A dlt pipeline gives each resource a table, and each batch of rows goes to your destination function. The destination on this page sends each batch to one Zerobus Ingest table, so the rows land in a Unity Catalog Delta table without a message bus between dlt and the lakehouse.
When to use this
- You already load data with dlt and want the same pipeline to write to a Delta table in Unity Catalog.
- You want loads to need only the Zerobus Ingest endpoint and a service principal, with no SQL warehouse connection at load time.
- Your data comes from a source that dlt can read, and you want the table to follow a fixed schema.
Zerobus Ingest doesn't change your table's schema. A record with a field the table doesn't have is rejected, so this page also covers how to change the table and the pipeline together. See Schema changes.
How it works
The pipeline has two parts. The resource yields one row at a time from your source, and the destination is a function that dlt calls with batches of 50 rows. For each batch, it opens a Zerobus Ingest stream to the table dlt_<resource name>, sends the rows as JSON, flushes the stream, and closes it. dlt keeps its pipeline state in a local directory. If Zerobus Ingest rejects a row, the function raises an error, the load job fails, and dlt keeps the load package for the next run.
The resource in Step 4 is a stand-in that generates sample rows. Any dlt source or resource that yields rows as dictionaries works with this destination, as long as its rows match the table. Arrow-producing sources are not covered. dlt's sources are listed in Additional resources.
dlt's internal _dlt_ columns and tables are not passed to the destination. The destination sends only the columns of your resource. Loads are append only: a resource with any other write disposition is refused with an error.
This example supports the column types in the Step 1 table: STRING (dlt text), top-level TIMESTAMP (sent as epoch microseconds), and DOUBLE. The freeze contract refuses a field that isn't in COLUMNS, such as a nested list. Other types, including DECIMAL, DATE, BINARY, VARIANT, and timestamps nested inside lists or dictionaries, are out of scope for this example.
Requirements
- A workspace with Unity Catalog in a supported region. The target table must meet the Zerobus Ingest table requirements.
- A service principal with an OAuth secret and the grants in Step 1. See Configure authentication.
- Databricks SQL to run the Step 1 statements. The pipeline itself doesn't use a SQL warehouse.
- Python 3.10 and above for dlt, and the Python version that the Zerobus Ingest SDK lists. Run the pipeline in a virtual environment.
Step 1: Create the table and grant access
Replace <catalog> and <schema> with your catalog and schema, then run this statement in Databricks SQL. The table has one column for each field the resource yields. The table name is dlt_ followed by the resource name, sensor_readings in this example.
CREATE TABLE <catalog>.<schema>.dlt_sensor_readings (
device_id STRING NOT NULL,
observed_at TIMESTAMP NOT NULL,
temperature_c DOUBLE,
humidity_pct DOUBLE
) USING DELTA;
Grant the service principal access. Replace <service-principal-uuid> with its application ID:
GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-uuid>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-uuid>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.dlt_sensor_readings TO `<service-principal-uuid>`;
Step 2: Install dlt and the Zerobus Ingest SDK
pip install "dlt>=1.30" databricks-zerobus-ingest-sdk
Step 3: Set the connection variables
Set these variables in the shell where you run the pipeline:
Variable | Value |
|---|---|
| The service principal's application ID ( |
| The service principal's OAuth secret |
| Your workspace URL, with |
| Your Zerobus Ingest endpoint, with |
| Your catalog and schema ( |
To find your values, see Get your workspace URL and Zerobus Ingest endpoint.
Your endpoint is https://<workspace-id>.zerobus.<region>.gcp.databricks.com.
Step 4: Create the pipeline
Save this as zerobus_pipeline.py.
"""Load rows from a dlt resource into Zerobus Ingest with a dlt custom destination.
The destination writes each batch through the Zerobus Ingest Python SDK, which gets and refreshes
the OAuth token itself.
"""
import os
from datetime import datetime, timedelta, timezone
import dlt
from zerobus.sdk.shared import TableProperties
from zerobus.sdk.sync import ZerobusSdk
# The columns of the Zerobus table dlt_sensor_readings. Create the table first, then keep this in step with it.
COLUMNS = {
"device_id": {"data_type": "text", "nullable": False},
"observed_at": {"data_type": "timestamp", "nullable": False},
"temperature_c": {"data_type": "double", "nullable": True},
"humidity_pct": {"data_type": "double", "nullable": True},
}
def pipeline_name() -> str:
"""The dlt pipeline name. Recovery imports it, so both use the same state."""
return f"zerobus_{os.environ['CATALOG']}_{os.environ['SCHEMA']}_sensor_readings"
@dlt.destination(batch_size=50, loader_file_format="typed-jsonl", name="zerobus")
def zerobus(items, table):
"""Write one batch of rows to the Zerobus table dlt_<resource name>. Append only."""
if table["write_disposition"] != "append":
raise ValueError(f"This destination supports append only; {table['name']} has write_disposition={table['write_disposition']!r}.")
sdk = ZerobusSdk(os.environ["ZEROBUS_SERVER_ENDPOINT"], os.environ["DATABRICKS_WORKSPACE_URL"])
target = f"{os.environ['CATALOG']}.{os.environ['SCHEMA']}.dlt_{table['name']}"
stream = sdk.create_stream(
os.environ["DATABRICKS_CLIENT_ID"], os.environ["DATABRICKS_CLIENT_SECRET"], TableProperties(target)
)
try:
for item in items:
# dlt passes TIMESTAMP values as datetime objects. Zerobus takes them as epoch microseconds.
stream.ingest_record_offset(
{k: round(v.timestamp() * 1_000_000) if isinstance(v, datetime) else v for k, v in item.items()}
)
stream.flush()
finally:
stream.close()
@dlt.resource(
name="sensor_readings",
write_disposition="append",
columns=COLUMNS,
max_table_nesting=0, # nested lists stay in one column, so the column freeze below stops them
schema_contract={"columns": "freeze"},
)
def sensor_readings(count: int = 300):
start = datetime(2026, 11, 16, tzinfo=timezone.utc)
for i in range(count):
yield {
"device_id": f"device-{i % 3}",
"observed_at": start + timedelta(minutes=i),
"temperature_c": 20.0 + (i % 17) * 0.5,
"humidity_pct": 40.0 + (i % 23) * 0.75,
}
if __name__ == "__main__":
pipeline = dlt.pipeline(pipeline_name=pipeline_name(), destination=zerobus)
print(pipeline.run(sensor_readings()))
The pipeline code works as follows:
@dlt.destinationturns the function into a dlt destination.loader_file_format="typed-jsonl"gives the function typed values, andbatch_size=50sets how many rows each call receives.COLUMNStells dlt the shape of the resource, so dlt doesn't infer one from the data. It must match the Zerobus Ingest table.max_table_nesting=0keeps a nested list in one column instead of creating a child table. Zerobus Ingest has no child table for dlt to write to.schema_contract={"columns": "freeze"}stops the load in dlt when the data has a column that isn't inCOLUMNS, including a nested list kept in one column. See Schema changes.- Zerobus Ingest takes a
TIMESTAMPas epoch microseconds (int64), per the type support table. An ISO 8601 string is rejected, so the destination converts eachdatetime. Python reads a naivedatetime(one without a timezone) in the host's local timezone, so yield timezone-aware values, as the example does. - The pipeline name is
zerobus_<catalog>_<schema>_sensor_readings, so each catalog, schema, and table has its own state. dlt keeps that state under its pipelines directory,~/.dlt/pipelines/<pipeline name>by default. SetDLT_DATA_DIRto move it. The name doesn't include the workspace. To use a different pipeline name for each workspace, changepipeline_name(). The recovery script imports the same function, so both keep the same state.
Step 5: Run the pipeline
python zerobus_pipeline.py
The last line reports the load package and its status. When the load succeeds, the status line reads is LOADED and contains no failed jobs.
Step 6: Query the table
SELECT device_id, observed_at, temperature_c, humidity_pct
FROM <catalog>.<schema>.dlt_sensor_readings
ORDER BY observed_at DESC, device_id
LIMIT 10;
Each run appends the rows again, so the same device and timestamp can appear more than one time. To keep one row per device and timestamp, select from the table with DISTINCT or QUALIFY, or write a MERGE into a second table.
Schema changes
The Zerobus Ingest table is the contract. dlt's default schema contract is evolve, which adds a new column to dlt's own schema and sends the new field to Zerobus Ingest. Zerobus Ingest doesn't add the column, so it rejects the record. This page's pipeline uses freeze to stop the load in dlt before anything is sent. The freeze covers new columns. This page doesn't set a data_type contract, so a value of a different type in a column that's already in COLUMNS isn't blocked by the freeze. That case is outside this example.
A failed load leaves its load package pending in dlt. dlt retries that package on the next run, with the columns it was extracted with. Changing COLUMNS doesn't change a package that is already pending, so discard the pending packages before you rerun.
Discarding removes the packages dlt extracted but hasn't loaded, along with the rows in them. dlt re-extracts those rows on the next run only if the source can replay them. For a source that can't replay (for example, events that are read one time and then gone), the discarded rows are lost. Batches of a discarded package that already reached Zerobus Ingest can also be written again. See the duplicate rows entry in Troubleshooting.
To add a field named battery_pct:
-
Add the column to the Zerobus Ingest table:
ALTER TABLE <catalog>.<schema>.dlt_sensor_readings ADD COLUMNS (battery_pct DOUBLE);. Adding a nullable column is a non-breaking change, so records that don't include it still load, and their value isNULL. -
Add the column to
COLUMNSin the pipeline and to the rows the resource yields. -
Discard the pending packages with this script. It uses the same pipeline name as
zerobus_pipeline.py. Run it one time from the directory that holdszerobus_pipeline.py:Python"""Abort the packages a failed load left pending, so the next run re-extracts with the current hints."""import dltfrom zerobus_pipeline import pipeline_name, zerobus # the name and destination defined in zerobus_pipeline.pypipeline = dlt.pipeline(pipeline_name=pipeline_name(), destination=zerobus)print("pending before:", pipeline.has_pending_data)pipeline.abort_packages()print("pending after:", pipeline.has_pending_data) -
Run the pipeline one time with
schema_contract={"columns": "evolve"}in the resource. This lets dlt add the column to its stored schema. Then set the contract back to{"columns": "freeze"}.
During that one run, the contract admits any new field the source sends, because dlt adds it to its schema. Make sure the source sends only the columns you planned to add. Zerobus Ingest rejects a field the table doesn't have, so the run fails. Batches written before the rejected record stay in the table.
If you skip step 3, the run fails again on the pending package, even after the table has the column. If you skip step 4, dlt stops on the new column, because its stored schema doesn't have it yet.
Troubleshooting
Symptom | Cause | Fix |
|---|---|---|
The load fails with | The source sent a column that isn't in | Follow Schema changes. Nothing reaches Zerobus Ingest, because dlt stops the load before the destination runs. |
The load fails with | The Zerobus Ingest table doesn't have the column | Add the column to the table, then follow Schema changes. |
The load fails with | A | Send epoch microseconds, as this page's destination does. |
The load fails with | The resource has a | Set |
The output says | dlt keeps the failed load package and retries it on the next run | Discard the package as in Schema changes before you rerun the pipeline. See the dlt documentation. |
Rows appear more than one time after a stop, a retry, or a discard | Zerobus Ingest provides at-least-once delivery, so a batch that was in flight when the run stopped can land twice. dlt doesn't resend the batches it committed: a run that stops partway resumes with the rest of the package. A batch that fails partway and is retried can also land twice. | Deduplicate in the query, as in Step 6. |
Additional resources
- Zerobus Ingest: the endpoint, the workspace URL, and authentication.
- Zerobus Ingest table requirements: what the target table must meet.
- Schema management: how Zerobus Ingest matches records to the table.
- dlt custom destinations: the decorator, batch sizes, and file formats used on this page.
- dlt sources and resources and verified sources: the data you can load with this destination.
- dlt schema contracts: the
evolve,freeze, anddiscard_rowmodes. - Zerobus Ingest Python SDK: the stream API, the record formats, and the authentication options.
- Zerobus Ingest errors: the error codes Zerobus Ingest returns.