RabbitMQ connector limitations

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page lists limitations and considerations for ingesting data from RabbitMQ using Databricks Lakeflow Connect.

Supported data

The RabbitMQ connector ingests messages from one or more RabbitMQ classic queues. You specify the queue to consume from using the queue connector option. Each queue is ingested into its own destination streaming table.

Connector-specific limitations

• Only the decoded message body and metadata columns are written to the destination table. Broker-internal fields beyond these are not exposed.
• The connector supports username and password authentication. Other authentication methods are not supported.
• Destination tables are append-only. Upserts and deletes are not supported. SCD type 1 and SCD type 2 are not supported because RabbitMQ classic queues do not provide a guaranteed unique message ID.
• Each pipeline uses a single Unity Catalog connection (one RabbitMQ broker). To ingest from multiple brokers, create separate pipelines.
• RabbitMQ Streams are not supported.
• The managed RabbitMQ connector runs exclusively on serverless compute. Classic compute pipelines are not supported.
• Duplicates can occur after failure recovery or message redelivery. Deduplicate downstream if needed.
• Per-queue FIFO order is best-effort with a single consumer. With the connector's parallel consumers, messages can arrive out of order.
• The pipeline runs continuously. Trigger.AvailableNow and Workflow-based scheduling are not supported.