Skip to main content
Last updated on

Troubleshoot the Kafka connector

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page shows how to troubleshoot common errors with the managed Kafka connector in Lakeflow Connect.

For behavioral questions (for example, what happens when records are deleted from Kafka before the pipeline reads them), see Kafka connector FAQs.

Stream returns no records even though data exists in the topic​

Cause: The starting_offset option defaults to latest, which only reads messages that arrive after the stream starts. If data was written to the topic before the pipeline began, those records are not read.

Resolution: Set starting_offset: earliest in your kafka_options to read from the beginning of the topic on the first run. Note that this setting applies only when no checkpoint exists. After the pipeline has run once, it continues from the last committed offset regardless of starting_offset.

Stream connects but no records arrive (authentication or ACL issue)​

Cause: The pipeline authenticated to the Kafka cluster successfully, but the credentials lack permission to read from the specified topic. Kafka ACLs may be blocking the consumer group or the topic.

Resolution:

  1. Verify that the username or service credential used in the Unity Catalog connection has READ permission on the topic in your Kafka ACL configuration.
  2. Check that the consumer group used by the pipeline is permitted. Contact your Kafka administrator to review ACL settings.
  3. Verify the topic name is correct — Kafka topic names are case-sensitive.

Pipeline fails to connect to the Kafka cluster (TimeoutException)​

Cause: The bootstrap servers are unreachable from the Databricks serverless compute environment. Common causes include network configuration, firewall rules, or an incorrect bootstrap server address in the connection.

Resolution:

  1. Verify the bootstrap server addresses in the Unity Catalog connection are correct and include the correct port (default: 9092).
  2. Ensure your Kafka cluster allows inbound connections from Databricks serverless compute. See Authentication for Kafka for network configuration guidance.
  3. If your cluster uses TLS, confirm the port matches the TLS listener (default: 9093).

Pipeline skips offsets after restarting​

Cause: This is expected behavior. If the pipeline falls behind and Kafka deletes records (due to topic retention settings) before the pipeline reads them, the connector automatically skips the missing offsets and continues from the next available offset. See Kafka connector FAQs for more information.

Resolution: To avoid skipping records, ensure your Kafka topic retention period is long enough to accommodate expected pipeline downtime or lag.

On this page