Skip to main content

Replicate data to Zerobus Ingest with Filament

info

The Databricks Zerobus Ingest sink is part of Filament, an open-source data movement engine. For the sink options, see Filament's Databricks Zerobus sink reference. Filament's connector overview marks the Databricks Zerobus sink as preview.

Maintained by: the Filament team at Galaxy, in the galaxy-io/filament repository.

This page shows how to replicate tables from a source that Filament supports to Delta tables in Unity Catalog. Filament copies data from a source into a sink, and its native Databricks Zerobus Ingest sink writes the rows into Unity Catalog Delta tables through Zerobus Ingest.

Full copies, and change data capture (CDC) where the source supports it, both work with the sink. Filament runs as a server, a control plane, and a worker, and it needs NATS JetStream as its event bus. The sink lives only in the worker's -libc image.

Choose a write mode​

Filament's sink writes in one of two modes. Pick one before you create the table, because each mode needs different columns.

Consideration

Full append

cdc_append

What lands

Every row of the source table on every run

Each change as its own row, with _filament_operation set to insert, update, or delete

Source requirement

Any source that Filament supports

A source that supports change data capture

Duplicates

Every run resends every row, so deduplicate on the key

Keys repeat by design, so deduplicate on the key, the source position, and the sequence

Choose it when

You want a full copy of each table, refreshed on every run

You want a history of inserts, updates, and deletes

Consideration

Full append

cdc_append

What lands

Every row of the source table on every run

Each change as its own row, with _filament_operation set to insert, update, or delete

Source requirement

Any source that Filament supports

A source that supports change data capture

Duplicates

Every run resends every row, so deduplicate on the key

Keys repeat by design, so deduplicate on the key, the source position, and the sequence

Choose it when

You want a full copy of each table, refreshed on every run

You want a history of inserts, updates, and deletes

When to use Filament​

Use Filament when you want tables from a supported source in Delta tables in Unity Catalog, and you already run or want to run Filament.

How it works​

  • Pipelines: A saved source, a saved sink, and a pipeline connect them. A pipeline sets the sync mode (full or cdc), the write mode (append), and the destination schema.
  • Sink: The Zerobus sink wraps the Zerobus Go SDK, a cgo wrapper over a Rust core. For each table it opens one Arrow Flight stream, sends batches of up to 50,000 rows or about 32 MiB, and flushes the stream when the run commits.
  • Write modes: The sink supports full append, incremental append, and cdc_append. It doesn't replace, upsert, or delete rows. A cdc_append table keeps every change as a row, with _filament_operation set to insert, update, or delete. The steps on this page use full append and cdc_append.
  • Lineage columns: Filament adds _filament_run_id and _filament_run_started_at to every destination table. cdc_append tables also get _filament_operation, _filament_source_position, and _filament_sequence.
  • Authentication: The sink authenticates with a service principal's OAuth client ID and secret, which must have the all-apis scope.
  • Delivery: Zerobus Ingest can deliver a row more than one time. A failed run can leave rows that a retry appends again. See Handle duplicates.

Requirements​

  • Filament with the databrickszerobus sink. The steps on this page use the 0.42.0 images and CLI.
  • The worker's -libc image for every worker that runs the sink. The standalone binary and the filament CLI don't include the sink.
  • Filament's own state database, which runs on PostgreSQL (PERSISTENCE_PROVIDER=postgres), as Filament's configuration guide describes. It is Filament's infrastructure, not a source.
  • NATS JetStream, which Filament requires as its event bus (NATS_URL).
  • Helm and a Kubernetes cluster for the deployment that Step 2 describes, and openssl to generate ENCRYPTION_KEY.
  • A source that Filament supports, from its supported sources list. Each source's page lists its requirements, including any keys and change-capture setup.
  • A workspace in a region that supports Zerobus Ingest. See Availability.
  • A service principal with an OAuth secret. See Configure authentication.
  • A SQL warehouse, if you let Filament create tables or set --sink-warehouse-id.
  • Target tables that meet the Zerobus Ingest table requirements.

Step 1: Get your endpoints and service principal​

The sink needs two values:

  • zerobus_endpoint: your Zerobus Ingest endpoint, without the https:// prefix.
  • workspace_url: your workspace URL, with the https:// prefix. The sink uses it to get OAuth tokens. Filament requires https://.
  • zerobus_endpoint: <workspace-id>.zerobus.<region>.gcp.databricks.com
  • workspace_url: https://<workspace-url>

To find your workspace ID, region, and URL, see Get your workspace URL and Zerobus Ingest endpoint.

Create a service principal for Filament in Settings, under Identity and access, and set these:

  • The Workspace access entitlement, which Zerobus Ingest requires. See Manage entitlements.
  • An OAuth secret with the all-apis scope. Filament's sink documentation notes that a secret without this scope fails at /oidc/v1/token with Scopes 'all-apis' are not assigned to the client.
  • Note the service principal's application ID. Filament calls it client_id.

Export the values in the shell you use for the CLI. Step 6 explains how the secret reaches the CLI:

sh
export DATABRICKS_CLIENT_ID=<application-id>
export DATABRICKS_CLIENT_SECRET=<client-secret>

Step 2: Deploy Filament​

Filament runs as three services, and each reads the same datastore, event bus, and secret settings:

  • The server serves the API and the web application.
  • The control plane schedules runs and dispatches them. Run exactly one replica.
  • The worker runs connector code, including the Zerobus sink.

Filament recommends Kubernetes with Helm for production. Follow Filament's Kubernetes guide to install the chart. It deploys the server, the control plane, a persistent worker, and a worker Job for each run.

Use the worker's -libc image for every worker in the chart, ghcr.io/galaxy-io/filament/worker:0.42.0-libc, because only that image includes the Zerobus sink. The persistent worker and the worker Jobs use the same image, so both need the -libc tag.

Set ENCRYPTION_KEY, which the configuration guide requires when the secret provider is the datastore, the default. Generate it with openssl rand -base64 32, and keep it. Changing it makes the stored secrets unreadable.

Turn on server authentication before you create any sink, except in a test deployment as Step 3 describes. The server stores the Databricks client secret and dispatches runs. With AUTH_PROVIDER unset, the API and web application are unauthenticated, so anyone who can reach the server can use its API, including the sink that holds the client secret. Set AUTH_PROVIDER to zitadel, keycloak, or proxy, with the variables that provider requires. Filament's configuration guide lists them in its Authentication section. With proxy, the gateway must strip any identity headers that a client sends, and it must be the only route to the server.

Under Kubernetes dispatch, Filament's runs and recovery guide describes a reaper that fails runs whose worker stops sending heartbeats. Troubleshooting lists the symptoms of a stuck run.

Step 3: Connect the CLI​

Install the filament CLI from the Install section of Filament's CLI guide. The guide provides an install script for Linux and macOS, Homebrew on macOS, and prebuilt binaries on GitHub Releases. Confirm the install with filament version.

Add a context for the server and select it. Use this path only for a test deployment with server authentication off. For any other deployment, Step 2 requires authentication, so use filament auth login instead.

sh
filament context add zb --server https://<filament-server>
filament context use zb
filament status

The CLI sends the server address, and Step 6 sends the Databricks client secret to this server. Use https://, with TLS on the ingress, load balancer, or tunnel in front of the server. Filament's Kubernetes guide notes that a NetworkPolicy doesn't encrypt the RPC connection, and recommends TLS or a service mesh to protect connection credentials in transit. Use plain http:// only for a server on the same host, such as a local test.

filament status shows whether the server is reachable. The server listens on port 8080 by default, as Filament's configuration guide describes. Turn on authentication before you create the sink, except in a test deployment, because the server holds the Databricks client secret and dispatches runs. Run filament auth login in place of filament context add, as the CLI guide describes. The CLI prompts for the client ID and secret of a Filament service account, not the Databricks service principal from Step 1. It verifies them, creates the context, and selects it.

Step 4: Create the target tables and grants​

Filament can create the schema and table for you, or you can create them yourself. The steps on this page create them yourself, so you control the columns. The table must match the source columns, with Filament's lineage columns after them.

The example source table orders has an id key, a customer column that is not null, and amount, status, and created_at columns. Match each column's type and nullability to the source:

SQL
CREATE TABLE <catalog>.<schema>.orders (
id BIGINT NOT NULL,
customer STRING NOT NULL,
amount DECIMAL(10,2),
status STRING,
created_at TIMESTAMP NOT NULL,
_filament_run_id STRING,
_filament_run_started_at TIMESTAMP
) USING DELTA;

Nullability must match the source. A non-nullable source column such as created_at must be NOT NULL in Delta, and a nullable source column must be nullable.

Grant the service principal access before the first run, as Filament's sink documentation requires. Grant USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT and MODIFY on each table, as Configure authentication describes. For the cdc_append table, grant the same privileges on its own schema and table.

To let Filament create the schema and tables, set --sink-create-table true on the pipeline in Step 7, and set --sink-warehouse-id on the sink. Filament runs CREATE SCHEMA IF NOT EXISTS and CREATE TABLE IF NOT EXISTS, then grants the service principal access to the objects it creates, as its sink documentation describes. For this path, the service principal also needs USE CATALOG and CREATE SCHEMA on the catalog, and the Can use permission on the SQL warehouse, as Filament's Databricks Zerobus sink reference describes.

create_table doesn't alter an existing table. With create_table on, a schema or table that already exists under another owner stops the sink from granting itself on it. The sink keeps going, because the grants may already be in place. If the stream then fails to open, the error lists the grants that failed.

Step 5: Create the source​

Choose a source from Filament's supported sources list. Its Sources table gives each source's ID, whether it's in preview, and its sync modes. Create the source with the ID as --source-connector, and set the connection fields from the source's page. filament source create --help lists the fields for a selected connector. Then check that Filament can see its tables:

sh
filament source create <source-name> --source-connector <source-id> <connection fields>
filament source discover <source-name>

For change data capture, check the source's page for the setup it needs.

Step 6: Create the sink​

Create the sink with the service principal's values from Step 1, the endpoint from Step 1, and your catalog. Set --sink-warehouse-id if you use create_table or want the sink to check the warehouse:

sh
filament sink create zb --sink-connector databrickszerobus --sink-workspace-url https://<workspace-url> --sink-zerobus-endpoint <zerobus-endpoint> --sink-client-id "$DATABRICKS_CLIENT_ID" --sink-client-secret "$DATABRICKS_CLIENT_SECRET" --sink-catalog <catalog> --sink-warehouse-id <warehouse-id>

Running the command this way puts the secret value on the command line, where other users of the host can see it in the process list while the command runs. Run it only on a trusted, single-user host, clear the shell history afterward, and rotate the secret if the host is shared. This is general guidance. Filament's documentation doesn't cover it. Filament can also resolve a named secret from its configured secret provider, which can be an external secret manager or its datastore, so the value doesn't have to go on the command line. See Filament's configuration guide for the secret provider settings.

Step 7: Create the pipeline and run it​

Create a full append pipeline or a cdc_append pipeline. Each one writes to the destination schema from Step 4:

sh
filament pipeline create orders-to-zb --source <source-name> --sink zb --resources orders --sync-mode full --write-mode append --sink-schema <schema>

Run the pipeline and list its runs:

sh
filament run orders-to-zb
filament run ls

A successful run reports its record count:

Output
Run <run-id> completed: <records> records, <bytes> bytes.

A full append pipeline sends every row on every run. The first run of a CDC pipeline takes a snapshot of the table, and later runs send the changes since the last run.

Step 8: Check the Delta table​

Query the table. Right after a run reports completed, the count can still show the earlier total for about 20 seconds, so wait that long before you query. After the first full run of the orders example, the table holds 20 rows with 20 distinct IDs.

SQL
SELECT count(*) AS rows_landed, count(DISTINCT id) AS distinct_ids
FROM <catalog>.<schema>.orders;

For a cdc_append table, count the operations:

SQL
SELECT _filament_operation, count(*) AS changes
FROM <catalog>.<cdc-schema>.payments
GROUP BY _filament_operation;

Change a source table's schema​

The sink doesn't drop, rename, or alter existing columns. Adding a nullable column is a non-breaking change, as Schema evolution describes. When you add a column to the source table, add it to the Delta table too, after the last source column and before Filament's lineage columns:

SQL
ALTER TABLE <catalog>.<schema>.orders ADD COLUMN coupon STRING;
ALTER TABLE <catalog>.<schema>.orders ALTER COLUMN coupon AFTER created_at;

If the source has a column that the Delta table doesn't, the run fails when its stream opens, and no rows are written. Add the Delta column, then run the pipeline again.

Handle duplicates​

Zerobus Ingest can deliver a row more than one time, so a retried run can append rows that were already written. The sink page describes this in Failure behavior.

  • Full append: every run resends every row. For example, if the first run appends 20 rows and a later run sends 21, the table holds 41 rows with 21 distinct IDs. Deduplicate on the primary key, for example with count(DISTINCT id) or a view that keeps one row per ID.
  • cdc_append: keys repeat by design, because each change is a row. Deduplicate on the key with _filament_source_position and _filament_sequence, and leave out the _filament_run_* columns, since they differ on a retry. To read the current state, keep the latest row per key by sorting on _filament_run_started_at, then _filament_sequence, and drop the key when that latest row is a delete.
  • Failed snapshots: if an initial snapshot fails, its rows stay in the table. The sink page explains how to tell which rows to ignore.

Troubleshooting​

Symptom

Cause

Fix

Schema comparison failed: Field 'created_at' is incompatible - 'created_at': nullability differs ... Error Code: 8001

The Delta column is nullable, but the source column is NOT NULL. The stream doesn't open.

Make the Delta column match the source, for example ALTER TABLE <catalog>.<schema>.orders ALTER COLUMN created_at SET NOT NULL.

Client field 'coupon' does not exist in Delta schema. Error Code: 8001

A source column has no matching column in the Delta table. The run fails when the stream opens, and no rows are written.

Add the column to the Delta table after the last source column. See Change a source table's schema.

Client field '_filament_run_id' is out of order: must follow Delta schema column ordering. Error Code: 8001

A new column comes after Filament's lineage columns.

Move the column with ALTER COLUMN <column> AFTER <last-source-column>. Add new columns AFTER the last source column.

run overlaps an active replication route

A run is still RUNNING after its worker stopped, and new runs of that pipeline are refused.

Under Kubernetes dispatch, Filament's reaper fails runs whose worker stops reporting heartbeats, per its runs and recovery guide. In-process dispatch has no reaper, so use Kubernetes dispatch (Step 2). Rows already written by a failed run stay in the table, so drop them by its _filament_run_id.

Scopes 'all-apis' are not assigned to the client

Per Filament's sink documentation, the OAuth secret lacks the all-apis scope.

Generate a new secret with the all-apis scope, and export it as DATABRICKS_CLIENT_SECRET. Then run filament sink edit zb --sink-client-secret "$DATABRICKS_CLIENT_SECRET". It changes only the fields you pass, so the other sink settings stay as they are.

Every token request fails

Per Filament's sink documentation, token requests fail without the Workspace access entitlement, which Zerobus Ingest requires. The service principal lacks it.

Add the Workspace access entitlement to the service principal in its settings.

401 or invalid_authorization_details

The service principal lacks USE SCHEMA on the schema or SELECT, MODIFY on the table.

Grant the privileges from Step 4, then run the pipeline again.

Connector databrickszerobus not found

The worker is not the -libc image, so the sink isn't compiled in.

Run the worker from the 0.42.0-libc image tag.

Symptom

Cause

Fix

Schema comparison failed: Field 'created_at' is incompatible - 'created_at': nullability differs ... Error Code: 8001

The Delta column is nullable, but the source column is NOT NULL. The stream doesn't open.

Make the Delta column match the source, for example ALTER TABLE <catalog>.<schema>.orders ALTER COLUMN created_at SET NOT NULL.

Client field 'coupon' does not exist in Delta schema. Error Code: 8001

A source column has no matching column in the Delta table. The run fails when the stream opens, and no rows are written.

Add the column to the Delta table after the last source column. See Change a source table's schema.

Client field '_filament_run_id' is out of order: must follow Delta schema column ordering. Error Code: 8001

A new column comes after Filament's lineage columns.

Move the column with ALTER COLUMN <column> AFTER <last-source-column>. Add new columns AFTER the last source column.

run overlaps an active replication route

A run is still RUNNING after its worker stopped, and new runs of that pipeline are refused.

Under Kubernetes dispatch, Filament's reaper fails runs whose worker stops reporting heartbeats, per its runs and recovery guide. In-process dispatch has no reaper, so use Kubernetes dispatch (Step 2). Rows already written by a failed run stay in the table, so drop them by its _filament_run_id.

Scopes 'all-apis' are not assigned to the client

Per Filament's sink documentation, the OAuth secret lacks the all-apis scope.

Generate a new secret with the all-apis scope, and export it as DATABRICKS_CLIENT_SECRET. Then run filament sink edit zb --sink-client-secret "$DATABRICKS_CLIENT_SECRET". It changes only the fields you pass, so the other sink settings stay as they are.

Every token request fails

Per Filament's sink documentation, token requests fail without the Workspace access entitlement, which Zerobus Ingest requires. The service principal lacks it.

Add the Workspace access entitlement to the service principal in its settings.

401 or invalid_authorization_details

The service principal lacks USE SCHEMA on the schema or SELECT, MODIFY on the table.

Grant the privileges from Step 4, then run the pipeline again.

Connector databrickszerobus not found

The worker is not the -libc image, so the sink isn't compiled in.

Run the worker from the 0.42.0-libc image tag.

Additional resources​

This page doesn't cover Kubernetes dispatch end to end or the incremental append mode.