Kafka connector limitations

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page lists limitations and considerations for ingesting data from Apache Kafka using Databricks Lakeflow Connect.

Supported data

The Kafka connector ingests data from one or more Kafka topics. You specify which topics to ingest using the topics or topic_pattern connector option. Any topic accessible through the Kafka connection can be ingested.

Connector-specific limitations

• Metadata columns not available: Only the key and value columns are written to the destination table. Kafka metadata columns — including topic, partition, offset, timestamp, timestampType, and headers — are not available in the destination table in Beta.
• Append-only tables: Destination tables are append-only. Upserts and deletes are not supported.
• One cluster per pipeline: Each pipeline uses a single Unity Catalog connection (one Kafka cluster). To ingest from multiple Kafka clusters, create separate pipelines.
• Serverless compute only: The managed Kafka connector runs exclusively on serverless compute. Classic compute pipelines are not supported.
• No UI-based pipeline authoring: Pipeline creation through the Databricks UI is not supported in Beta. Use Declarative Automation Bundles or a Databricks notebook.
• No column selection or deselection: Selecting or deselecting specific columns is not supported in Beta.
• No row filtering: Row-level filtering is not supported.
• No SCD type 2: Because destination tables are append-only, SCD type 2 history tracking is not supported.
• No schema evolution for renamed columns: Column renames are not tracked. A renamed column in the source is treated as a new column in the destination.
• No orchestration using Databricks Workflows: The pipeline runs continuously. Workflow-based scheduling is not supported.

Fanout limitations

Preview

This feature is in Private Preview. To try it, reach out to your Databricks contact.

The following limitations apply when you route records to multiple tables using fanout. See Route records to multiple tables (fanout) to configure fanout.

• Configure fanout_options on a schema object, not a table object. Do not set source_catalog or source_schema on the fanout schema object.
• Each record routes to exactly one destination table, determined by its fanout_by value. Fanning a single record out to multiple tables isn't supported.
• Fanout requires a continuous pipeline (continuous: true). Triggered (one-time) pipelines aren't supported.
• You can apply at most one transform to each route, and it must use format: JSON. AVRO and PROTOBUF transforms are not supported for fanout.
• The fanout_by expression must resolve to a STRING, reference at least one input column, and be deterministic. User-defined functions (UDFs) are not supported. The routing key must not be null.
• The resolved fanout_by value is used as the destination table-name segment verbatim, without quoting or sanitization, so it must be a valid unquoted table identifier. Values with spaces, dots, or other characters that aren't valid in an unquoted identifier fail the write, as does a value that is entirely digits (for example, 123). A leading digit is allowed when the value also contains a letter or underscore, such as 2024_events.
• fanout_by is evaluated against the raw source record (with the Kafka key and value columns). The value column is binary, so cast it to a string before extracting a field (for example, cast(value as string):event_type::string). A fanout_options transform runs after routing, so it can't produce a column that fanout_by references.