Community connectors in Lakeflow Connect

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Community connectors are open-source connectors that extend Lakeflow Connect to sources without managed connector support. The community builds and maintains them. You have the following options:

• • Use a registered community connector
  • Use a registered community connector to ingest data from a supported source into Databricks.
• • Create a custom connector
  • Create a custom connector for a new source using the community framework and templates.

How community connectors work

Community connectors are built on the LakeflowConnect interface, which wraps the Spark Python Data Source API. Each connector handles authentication, schema discovery, and incremental data reads so you can create, configure, and run an ingestion pipeline backed by Lakeflow Spark Declarative Pipelines.

When you use a community connector, Databricks clones the connector source code from a GitHub repository into a workspace directory that you specify. The pipeline then reads the connector source code at runtime and executes the ingestion logic against the configured source.

Supported sources

The community regularly registers new connectors. For the latest list of supported sources, see the Add data UI (Data Ingestion) in your Databricks workspace or the Lakeflow Community Connectors repository on GitHub.

Considerations

• Community connectors are under active development. Interfaces and behavior are subject to change.
• Databricks doesn't maintain community connectors. They're not backed by Databricks SLAs and don't guarantee forward compatibility.

Submit feedback

Report bugs and submit feature requests in the Lakeflow Community Connectors repository.