Replicate data to Zerobus Ingest with Filament
The Databricks Zerobus Ingest sink is part of Filament, an open-source data movement engine. For the sink options, see Filament's Databricks Zerobus sink reference. Filament's connector overview marks the Databricks Zerobus sink as preview.
Maintained by: the Filament team at Galaxy, in the galaxy-io/filament repository.
This page shows how to replicate tables from a source that Filament supports to Delta tables in Unity Catalog. Filament copies data from a source into a sink, and its native Databricks Zerobus Ingest sink writes the rows into Unity Catalog Delta tables through Zerobus Ingest.
Full copies, and change data capture (CDC) where the source supports it, both work with the sink. Filament runs as a server, a control plane, and a worker, and it needs NATS JetStream as its event bus. The sink lives only in the worker's -libc image.
Choose a write mode
Filament's sink writes in one of two modes. Pick one before you create the table, because each mode needs different columns.
Consideration | Full append | cdc_append |
|---|---|---|
What lands | Every row of the source table on every run | Each change as its own row, with |
Source requirement | Any source that Filament supports | A source that supports change data capture |
Duplicates | Every run resends every row, so deduplicate on the key | Keys repeat by design, so deduplicate on the key, the source position, and the sequence |
Choose it when | You want a full copy of each table, refreshed on every run | You want a history of inserts, updates, and deletes |
When to use Filament
Use Filament when you want tables from a supported source in Delta tables in Unity Catalog, and you already run or want to run Filament.
How it works
- Pipelines: A saved source, a saved sink, and a pipeline connect them. A pipeline sets the sync mode (
fullorcdc), the write mode (append), and the destination schema. - Sink: The Zerobus sink wraps the Zerobus Go SDK, a cgo wrapper over a Rust core. For each table it opens one Arrow Flight stream, sends batches of up to 50,000 rows or about 32 MiB, and flushes the stream when the run commits.
- Write modes: The sink supports full append, incremental append, and
cdc_append. It doesn't replace, upsert, or delete rows. Acdc_appendtable keeps every change as a row, with_filament_operationset toinsert,update, ordelete. The steps on this page use full append andcdc_append. - Lineage columns: Filament adds
_filament_run_idand_filament_run_started_atto every destination table.cdc_appendtables also get_filament_operation,_filament_source_position, and_filament_sequence. - Authentication: The sink authenticates with a service principal's OAuth client ID and secret, which must have the
all-apisscope. - Delivery: Zerobus Ingest can deliver a row more than one time. A failed run can leave rows that a retry appends again. See Handle duplicates.
Requirements
- Filament with the
databrickszerobussink. The steps on this page use the 0.42.0 images and CLI. - The worker's
-libcimage for every worker that runs the sink. The standalone binary and thefilamentCLI don't include the sink. - Filament's own state database, which runs on PostgreSQL (
PERSISTENCE_PROVIDER=postgres), as Filament's configuration guide describes. It is Filament's infrastructure, not a source. - NATS JetStream, which Filament requires as its event bus (
NATS_URL). - Helm and a Kubernetes cluster for the deployment that Step 2 describes, and
opensslto generateENCRYPTION_KEY. - A source that Filament supports, from its supported sources list. Each source's page lists its requirements, including any keys and change-capture setup.
- A workspace in a region that supports Zerobus Ingest. See Availability.
- A service principal with an OAuth secret. See Configure authentication.
- A SQL warehouse, if you let Filament create tables or set
--sink-warehouse-id. - Target tables that meet the Zerobus Ingest table requirements.
Step 1: Get your endpoints and service principal
The sink needs two values:
zerobus_endpoint: your Zerobus Ingest endpoint, without thehttps://prefix.workspace_url: your workspace URL, with thehttps://prefix. The sink uses it to get OAuth tokens. Filament requireshttps://.
zerobus_endpoint:<workspace-id>.zerobus.<region>.gcp.databricks.comworkspace_url:https://<workspace-url>
To find your workspace ID, region, and URL, see Get your workspace URL and Zerobus Ingest endpoint.
Create a service principal for Filament in Settings, under Identity and access, and set these:
- The Workspace access entitlement, which Zerobus Ingest requires. See Manage entitlements.
- An OAuth secret with the
all-apisscope. Filament's sink documentation notes that a secret without this scope fails at/oidc/v1/tokenwithScopes 'all-apis' are not assigned to the client. - Note the service principal's application ID. Filament calls it
client_id.
Export the values in the shell you use for the CLI. Step 6 explains how the secret reaches the CLI:
export DATABRICKS_CLIENT_ID=<application-id>
export DATABRICKS_CLIENT_SECRET=<client-secret>
Step 2: Deploy Filament
Filament runs as three services, and each reads the same datastore, event bus, and secret settings:
- The server serves the API and the web application.
- The control plane schedules runs and dispatches them. Run exactly one replica.
- The worker runs connector code, including the Zerobus sink.
Filament recommends Kubernetes with Helm for production. Follow Filament's Kubernetes guide to install the chart. It deploys the server, the control plane, a persistent worker, and a worker Job for each run.
Use the worker's -libc image for every worker in the chart, ghcr.io/galaxy-io/filament/worker:0.42.0-libc, because only that image includes the Zerobus sink. The persistent worker and the worker Jobs use the same image, so both need the -libc tag.
Set ENCRYPTION_KEY, which the configuration guide requires when the secret provider is the datastore, the default. Generate it with openssl rand -base64 32, and keep it. Changing it makes the stored secrets unreadable.
Turn on server authentication before you create any sink, except in a test deployment as Step 3 describes. The server stores the Databricks client secret and dispatches runs. With AUTH_PROVIDER unset, the API and web application are unauthenticated, so anyone who can reach the server can use its API, including the sink that holds the client secret. Set AUTH_PROVIDER to zitadel, keycloak, or proxy, with the variables that provider requires. Filament's configuration guide lists them in its Authentication section. With proxy, the gateway must strip any identity headers that a client sends, and it must be the only route to the server.
Under Kubernetes dispatch, Filament's runs and recovery guide describes a reaper that fails runs whose worker stops sending heartbeats. Troubleshooting lists the symptoms of a stuck run.
Step 3: Connect the CLI
Install the filament CLI from the Install section of Filament's CLI guide. The guide provides an install script for Linux and macOS, Homebrew on macOS, and prebuilt binaries on GitHub Releases. Confirm the install with filament version.
Add a context for the server and select it. Use this path only for a test deployment with server authentication off. For any other deployment, Step 2 requires authentication, so use filament auth login instead.
filament context add zb --server https://<filament-server>
filament context use zb
filament status
The CLI sends the server address, and Step 6 sends the Databricks client secret to this server. Use https://, with TLS on the ingress, load balancer, or tunnel in front of the server. Filament's Kubernetes guide notes that a NetworkPolicy doesn't encrypt the RPC connection, and recommends TLS or a service mesh to protect connection credentials in transit. Use plain http:// only for a server on the same host, such as a local test.
filament status shows whether the server is reachable. The server listens on port 8080 by default, as Filament's configuration guide describes. Turn on authentication before you create the sink, except in a test deployment, because the server holds the Databricks client secret and dispatches runs. Run filament auth login in place of filament context add, as the CLI guide describes. The CLI prompts for the client ID and secret of a Filament service account, not the Databricks service principal from Step 1. It verifies them, creates the context, and selects it.
Step 4: Create the target tables and grants
Filament can create the schema and table for you, or you can create them yourself. The steps on this page create them yourself, so you control the columns. The table must match the source columns, with Filament's lineage columns after them.
- Full append
- Cdc_append
The example source table orders has an id key, a customer column that is not null, and amount, status, and created_at columns. Match each column's type and nullability to the source:
CREATE TABLE <catalog>.<schema>.orders (
id BIGINT NOT NULL,
customer STRING NOT NULL,
amount DECIMAL(10,2),
status STRING,
created_at TIMESTAMP NOT NULL,
_filament_run_id STRING,
_filament_run_started_at TIMESTAMP
) USING DELTA;
Nullability must match the source. A non-nullable source column such as created_at must be NOT NULL in Delta, and a nullable source column must be nullable.
The example source table payments has an id key, and order_id, amount, state, and updated_at columns that are not null in the source. A cdc_append table holds change history, so every column is nullable, including the key. Put it in a schema of its own. Filament names the table after the resource, and the sink documentation points a cdc_append pipeline at its own schema:
CREATE TABLE <catalog>.<cdc-schema>.payments (
id BIGINT,
order_id BIGINT,
amount DECIMAL(10,2),
state STRING,
updated_at TIMESTAMP,
_filament_run_id STRING,
_filament_run_started_at TIMESTAMP,
_filament_operation STRING,
_filament_source_position STRING,
_filament_sequence BIGINT
) USING DELTA;
Grant the service principal access before the first run, as Filament's sink documentation requires. Grant USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT and MODIFY on each table, as Configure authentication describes. For the cdc_append table, grant the same privileges on its own schema and table.
To let Filament create the schema and tables, set --sink-create-table true on the pipeline in Step 7, and set --sink-warehouse-id on the sink. Filament runs CREATE SCHEMA IF NOT EXISTS and CREATE TABLE IF NOT EXISTS, then grants the service principal access to the objects it creates, as its sink documentation describes. For this path, the service principal also needs USE CATALOG and CREATE SCHEMA on the catalog, and the Can use permission on the SQL warehouse, as Filament's Databricks Zerobus sink reference describes.
create_table doesn't alter an existing table. With create_table on, a schema or table that already exists under another owner stops the sink from granting itself on it. The sink keeps going, because the grants may already be in place. If the stream then fails to open, the error lists the grants that failed.
Step 5: Create the source
Choose a source from Filament's supported sources list. Its Sources table gives each source's ID, whether it's in preview, and its sync modes. Create the source with the ID as --source-connector, and set the connection fields from the source's page. filament source create --help lists the fields for a selected connector. Then check that Filament can see its tables:
filament source create <source-name> --source-connector <source-id> <connection fields>
filament source discover <source-name>
For change data capture, check the source's page for the setup it needs.
Step 6: Create the sink
Create the sink with the service principal's values from Step 1, the endpoint from Step 1, and your catalog. Set --sink-warehouse-id if you use create_table or want the sink to check the warehouse:
filament sink create zb --sink-connector databrickszerobus --sink-workspace-url https://<workspace-url> --sink-zerobus-endpoint <zerobus-endpoint> --sink-client-id "$DATABRICKS_CLIENT_ID" --sink-client-secret "$DATABRICKS_CLIENT_SECRET" --sink-catalog <catalog> --sink-warehouse-id <warehouse-id>
Running the command this way puts the secret value on the command line, where other users of the host can see it in the process list while the command runs. Run it only on a trusted, single-user host, clear the shell history afterward, and rotate the secret if the host is shared. This is general guidance. Filament's documentation doesn't cover it. Filament can also resolve a named secret from its configured secret provider, which can be an external secret manager or its datastore, so the value doesn't have to go on the command line. See Filament's configuration guide for the secret provider settings.
Step 7: Create the pipeline and run it
Create a full append pipeline or a cdc_append pipeline. Each one writes to the destination schema from Step 4:
- Full append
- Cdc_append
filament pipeline create orders-to-zb --source <source-name> --sink zb --resources orders --sync-mode full --write-mode append --sink-schema <schema>
filament pipeline create payments-cdc-to-zb --source <cdc-source-name> --sink zb --resources payments --sync-mode cdc --write-mode append --sink-schema <cdc-schema>
Add --sink-create-table true to this command if you want Filament to create the table instead of creating it yourself in Step 4.
Run the pipeline and list its runs:
filament run orders-to-zb
filament run ls
A successful run reports its record count:
Run <run-id> completed: <records> records, <bytes> bytes.
A full append pipeline sends every row on every run. The first run of a CDC pipeline takes a snapshot of the table, and later runs send the changes since the last run.
Step 8: Check the Delta table
Query the table. Right after a run reports completed, the count can still show the earlier total for about 20 seconds, so wait that long before you query. After the first full run of the orders example, the table holds 20 rows with 20 distinct IDs.
SELECT count(*) AS rows_landed, count(DISTINCT id) AS distinct_ids
FROM <catalog>.<schema>.orders;
For a cdc_append table, count the operations:
SELECT _filament_operation, count(*) AS changes
FROM <catalog>.<cdc-schema>.payments
GROUP BY _filament_operation;
Change a source table's schema
The sink doesn't drop, rename, or alter existing columns. Adding a nullable column is a non-breaking change, as Schema evolution describes. When you add a column to the source table, add it to the Delta table too, after the last source column and before Filament's lineage columns:
ALTER TABLE <catalog>.<schema>.orders ADD COLUMN coupon STRING;
ALTER TABLE <catalog>.<schema>.orders ALTER COLUMN coupon AFTER created_at;
If the source has a column that the Delta table doesn't, the run fails when its stream opens, and no rows are written. Add the Delta column, then run the pipeline again.
Handle duplicates
Zerobus Ingest can deliver a row more than one time, so a retried run can append rows that were already written. The sink page describes this in Failure behavior.
- Full append: every run resends every row. For example, if the first run appends 20 rows and a later run sends 21, the table holds 41 rows with 21 distinct IDs. Deduplicate on the primary key, for example with
count(DISTINCT id)or a view that keeps one row per ID. - cdc_append: keys repeat by design, because each change is a row. Deduplicate on the key with
_filament_source_positionand_filament_sequence, and leave out the_filament_run_*columns, since they differ on a retry. To read the current state, keep the latest row per key by sorting on_filament_run_started_at, then_filament_sequence, and drop the key when that latest row is a delete. - Failed snapshots: if an initial snapshot fails, its rows stay in the table. The sink page explains how to tell which rows to ignore.
Troubleshooting
Symptom | Cause | Fix |
|---|---|---|
| The Delta column is nullable, but the source column is | Make the Delta column match the source, for example |
| A source column has no matching column in the Delta table. The run fails when the stream opens, and no rows are written. | Add the column to the Delta table after the last source column. See Change a source table's schema. |
| A new column comes after Filament's lineage columns. | Move the column with |
| A run is still | Under Kubernetes dispatch, Filament's reaper fails runs whose worker stops reporting heartbeats, per its runs and recovery guide. In-process dispatch has no reaper, so use Kubernetes dispatch (Step 2). Rows already written by a failed run stay in the table, so drop them by its |
| Per Filament's sink documentation, the OAuth secret lacks the | Generate a new secret with the |
Every token request fails | Per Filament's sink documentation, token requests fail without the Workspace access entitlement, which Zerobus Ingest requires. The service principal lacks it. | Add the Workspace access entitlement to the service principal in its settings. |
| The service principal lacks | Grant the privileges from Step 4, then run the pipeline again. |
| The worker is not the | Run the worker from the |
Additional resources
- Filament's Databricks Zerobus sink reference: the configuration fields, type mapping, and write modes.
- Filament's Kubernetes guide: the recommended production deployment, with the Helm chart.
- Filament's configuration guide: every server, control plane, and worker variable.
- Filament's supported sources list: every source, with its preview status and sync modes.
- Filament's runs and recovery guide: how runs fail, resume, and recover.
example_clients/filament_zerobusin the zerobus-ingest-examples repository: an example client that uses the same tables as these steps.- Schema management: plan table schema changes.
This page doesn't cover Kubernetes dispatch end to end or the incremental append mode.