Skip to main content
Last updated on

Kafka connector FAQs

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page contains answers to frequently asked questions about the managed Kafka ingestion connector in Databricks Lakeflow Connect.

General managed connector FAQs​

See Managed connector FAQs for FAQs that apply to all Lakeflow Connect managed connectors. The following are specific to the Kafka connector.

Connector-specific FAQs​

Why am I getting a TimeoutException when connecting to Kafka?​

Common causes include:

  • Network connectivity: The serverless compute cannot reach the Kafka brokers. Check firewall rules, security groups, and VPC configurations.
  • Wrong bootstrap servers: Verify the bootstrap server hostname and port in your connection are correct.
  • DNS resolution: Ensure the Kafka broker hostnames can be resolved from the Databricks network.
  • SSL/TLS issues: If using SSL, verify certificates are correctly configured.

For Private Link or VPC peering setups, ensure the correct network routes are in place. See Authentication for authentication configuration details.

Can I read from multiple Kafka topics in a single pipeline?​

Yes. You can subscribe to multiple topics in the same pipeline table definition using either:

  • topics: Provide a list of topic names, for example topics: [topic1, topic2].
  • topic_pattern: Use a Java regular expression to match topic names, for example topic_pattern: "topic-.*". topic_pattern is mutually exclusive with topics.

Both fields are set under connector_options.kafka_options. See Kafka connector reference for the full options reference.

Why is my stream returning no records even though data exists in the topic?​

Common causes include:

  • Wrong starting_offset setting: The default value is latest, which only reads new data arriving after the stream starts. Set starting_offset to earliest to read existing data.
  • Wrong topic name: Verify you're subscribing to the correct topic. Kafka topic names are case-sensitive.
  • Authentication issues: Your stream may have connected successfully but lacks permission to read from the topic. Check your Kafka ACLs.

How can I monitor how far behind my stream is from the latest Kafka offsets?​

See View Streaming Metrics. The streaming metrics UI shows the offset lag per partition, which indicates how far behind the pipeline is from the latest available offsets in the topic.

What happens if records are deleted from Kafka before the pipeline reads them?​

If the pipeline falls behind and Kafka deletes records before they are read (due to topic retention settings), the pipeline automatically skips the missing offsets and continues from the next available offset.

This behavior is intentional. Stopping the pipeline on a retention gap would cause the pipeline to fall further behind while waiting for intervention, increasing the total amount of data missed. By continuing past the gap, the pipeline resumes reading new records as quickly as possible.

If your use case requires guaranteeing that every record is processed, ensure your Kafka topic retention period is long enough to accommodate any expected pipeline downtime or lag.

On this page