Kafka connector

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

The managed Kafka connector in Lakeflow Connect allows you to ingest data from Apache Kafka topics into streaming tables in Databricks.

Kafka connector components

Component	Description
Connection	A Unity Catalog securable object that stores the bootstrap server address and authentication credentials for your Kafka cluster. The managed Kafka connector uses this connection to authenticate without requiring credentials in your pipeline configuration.
Ingestion pipeline	A pipeline that continuously reads from one or more Kafka topics and writes the results to streaming tables. The pipeline runs on serverless compute.
Destination tables	The streaming tables where the ingestion pipeline writes the data.

The following table provides an overview of the end-to-end Kafka ingestion flow, based on user type:

User	Steps
Admin	Use Catalog Explorer to create a connection so that any user with `USE CONNECTION` or `ALL PRIVILEGES` can create pipelines. See Create a Kafka connection.
Non-admin	Use any supported interface to create a pipeline from an existing connection. See Ingest data from Apache Kafka.

Feature	Availability
UI-based pipeline authoring	Not supported (Beta)
API-based pipeline authoring	Supported
Declarative Automation Bundles	Supported
Continuous streaming ingestion	Supported
Unity Catalog governance	Supported
Orchestration using Databricks Workflows	Not supported
SCD type 2	Not supported
Column selection and deselection	Not supported (Beta)
Row filtering	Not supported
Automated schema evolution: New columns	Supported
Automated schema evolution: Data type changes	Not supported
Automated schema evolution: Column renames	Not supported