Skip to main content
Last updated on

Troubleshoot the RabbitMQ connector

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page shows how to troubleshoot common errors with the managed RabbitMQ connector in Lakeflow Connect.

For behavioral questions (for example, what happens to messages when the pipeline crashes before acknowledging them), see RabbitMQ connector FAQs.

Pipeline fails to connect to the RabbitMQ broker (connection timeout)​

Cause: The broker is unreachable from the Databricks serverless compute environment. Common causes include an incorrect host or port in the connection, or blocked network connectivity.

Resolution:

  1. Verify the host and port in the Unity Catalog connection are correct. The default RabbitMQ port is 5672.
  2. Verify the virtual host in the connection is correct. The default virtual host is /.
  3. Ensure network connectivity from the serverless compute plane to your broker is allowed. Verify your VPC peering, security group, or firewall rules.

Stream returns no records even though data exists​

Cause: The pipeline connected successfully but is not receiving messages. Common causes include a wrong queue name, a competing external consumer, or messages that were already consumed and acknowledged by another consumer.

Resolution:

  1. Verify the queue name in your rabbitmq_options matches the queue on the broker.
  2. Check whether another (non-Spark) consumer is reading from the same queue. RabbitMQ delivers each message to exactly one consumer.
  3. Confirm that the connection user has read permission on the queue.

Stream connects but no records arrive (authentication or authorization issue)​

Cause: The pipeline authenticated to the broker successfully, but the credentials lack permission to consume from the specified queue.

Resolution: Verify that the username used in the Unity Catalog connection has read permission on the queue. Contact your RabbitMQ administrator to review permissions.

Messages are redelivered after a restart​

Cause: This is expected behavior. RabbitMQ provides at-least-once delivery. If the pipeline crashes or disconnects before acknowledging messages, those messages remain in the queue and are redelivered on reconnect. Redelivered messages have _redelivered set to true.

Resolution: If your use case requires exactly-once processing, deduplicate downstream using an application-specific key. See Can the connector guarantee exactly-once delivery?.

On this page