OpenAI connector

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

The managed OpenAI connector in Lakeflow Connect ingests organization administration data, such as users, projects, API keys, usage, costs, and audit logs, from OpenAI into Databricks.

Feature availability

Feature	Availability
UI-based pipeline authoring	Supported
API-based pipeline authoring	Supported
Declarative Automation Bundles	Supported
Incremental ingestion	Supported
Unity Catalog governance	Supported
Orchestration using Databricks Workflows	Supported
API-based column selection and deselection	Supported
API-based row filtering	Not supported
SCD type 2	Supported Applies to the mutable entity tables. The append-only audit_logs, usage_*, and costs tables record immutable events and daily buckets.
Automated schema evolution: New and deleted columns	Supported
Automated schema evolution: Data type changes	Not supported
Automated schema evolution: Column renames	Not supported Requires a full refresh.

Feature	Availability
UI-based pipeline authoring	Supported
API-based pipeline authoring	Supported
Declarative Automation Bundles	Supported
Incremental ingestion	Supported
Unity Catalog governance	Supported
Orchestration using Databricks Workflows	Supported
API-based column selection and deselection	Supported
API-based row filtering	Not supported
SCD type 2	Supported Applies to the mutable entity tables. The append-only audit_logs, usage_*, and costs tables record immutable events and daily buckets.
Automated schema evolution: New and deleted columns	Supported
Automated schema evolution: Data type changes	Not supported
Automated schema evolution: Column renames	Not supported Requires a full refresh.

Authentication methods

Authentication method	Availability
OAuth U2M	Not supported
OAuth M2M	Not supported
Basic authentication (username/password)	Not supported
Basic authentication (API key)	Supported OpenAI Admin API key.

Authentication method	Availability
OAuth U2M	Not supported
OAuth M2M	Not supported
Basic authentication (username/password)	Not supported
Basic authentication (API key)	Supported OpenAI Admin API key.

What to know before you start

Topic	Why it matters
Databricks user persona	The workflow depends on your Databricks user persona: Single-user: An admin user creates a Unity Catalog connection and an ingestion pipeline. Multi-user: An admin user creates a connection for non-admin users to create pipelines with.
Authentication method	The steps to create a connection depend on the authentication method you choose.
Interface	The steps to create a pipeline depend on the interface.
Ingestion frequency	The pipeline schedule depends on your latency and cost requirements.
Common patterns	Depending on your ingestion needs, the pipeline might use configurations like history tracking, column selection, and row filtering. Supported configurations vary by connector. See Feature availability.

Topic	Why it matters
Databricks user persona	The workflow depends on your Databricks user persona: Single-user: An admin user creates a Unity Catalog connection and an ingestion pipeline. Multi-user: An admin user creates a connection for non-admin users to create pipelines with.
Authentication method	The steps to create a connection depend on the authentication method you choose.
Interface	The steps to create a pipeline depend on the interface.
Ingestion frequency	The pipeline schedule depends on your latency and cost requirements.
Common patterns	Depending on your ingestion needs, the pipeline might use configurations like history tracking, column selection, and row filtering. Supported configurations vary by connector. See Feature availability.

Ingest from OpenAI in 3 steps

1. Configure OpenAI for ingestion (Admins) — Set up OpenAI to authenticate with Databricks.
2. Create a Unity Catalog connection (Admins) — Create a connection in Catalog Explorer to store credentials for Databricks to authenticate with OpenAI.
3. Create an ingestion pipeline (Admins or non-admins) — Select any supported interface and create a pipeline from an existing connection.