RabbitMQ connector
This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.
The managed RabbitMQ connector in Lakeflow Connect allows you to stream messages from RabbitMQ classic queues into streaming tables in Databricks. The connector provides at-least-once delivery semantics and handles authentication, message decoding, acknowledgement, and pipeline lifecycle management, so you don't need to write Structured Streaming code directly.
RabbitMQ connector components
Component | Description |
|---|---|
Connection | A Unity Catalog securable object that stores the broker endpoint and authentication credentials for your RabbitMQ broker. The managed RabbitMQ connector uses this connection to authenticate without requiring credentials in your pipeline configuration. |
Ingestion pipeline | A pipeline that continuously consumes messages from one or more RabbitMQ classic queues and writes the results to streaming tables. The pipeline runs on serverless compute. |
Destination tables | The streaming tables where the ingestion pipeline writes the data. |
Start ingesting from RabbitMQ
The following table provides an overview of the end-to-end RabbitMQ ingestion flow, based on user type:
User | Steps |
|---|---|
Admin | Use Catalog Explorer to create a connection so that any user with |
Non-admin | Use any supported interface to create a pipeline from an existing connection. See Ingest data from RabbitMQ. |
Feature availability
Feature | Availability |
|---|---|
UI-based pipeline authoring |
|
API-based pipeline authoring |
|
Declarative Automation Bundles |
|
Continuous streaming ingestion |
|
Unity Catalog governance |
|
Orchestration using Databricks Workflows |
|
SCD type 2 |
|
Column selection and deselection |
|
Row filtering |
|
Not supported (Beta)
SupportedAuthentication methods
Authentication method | Availability |
|---|---|
Username and password |
|
Service Credential |
|
OAuth U2M |
|
OAuth M2M |
|