Kafka connector limitations

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page lists limitations and considerations for ingesting data from Apache Kafka using Databricks Lakeflow Connect.

Supported data

The Kafka connector ingests data from one or more Kafka topics. You specify which topics to ingest using the topics or topic_pattern connector option. Any topic accessible through the Kafka connection can be ingested.

Connector-specific limitations

• Metadata columns not available: Only the key and value columns are written to the destination table. Kafka metadata columns — including topic, partition, offset, timestamp, timestampType, and headers — are not available in the destination table in Beta.
• Append-only tables: Destination tables are append-only. Upserts and deletes are not supported.
• One cluster per pipeline: Each pipeline uses a single Unity Catalog connection (one Kafka cluster). To ingest from multiple Kafka clusters, create separate pipelines.
• Serverless compute only: The managed Kafka connector runs exclusively on serverless compute. Classic compute pipelines are not supported.
• No UI-based pipeline authoring: Pipeline creation through the Databricks UI is not supported in Beta. Use Declarative Automation Bundles or a Databricks notebook.
• No column selection or deselection: Selecting or deselecting specific columns is not supported in Beta.
• No row filtering: Row-level filtering is not supported.
• No SCD type 2: Because destination tables are append-only, SCD type 2 history tracking is not supported.
• No schema evolution for renamed columns: Column renames are not tracked. A renamed column in the source is treated as a new column in the destination.
• No orchestration using Databricks Workflows: The pipeline runs continuously. Workflow-based scheduling is not supported.