Skip to main content
Last updated on

OpenAI connector limitations

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page contains information about known limitations of the managed OpenAI connector in Lakeflow Connect.

General software as a service (SaaS) connector limitations​

The limitations in this section apply to all SaaS connectors in Lakeflow Connect.

  • When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
  • When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
  • During source maintenance periods, Databricks might not be able to access your data.
  • If a source table name conflicts with an existing destination table name, the pipeline update fails.
  • Multi-destination pipeline support is API-only.
  • You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
  • If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
  • Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
  • The source system assumes that the cursor columns are monotonically increasing.
  • The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Connector-specific limitations​

The limitations in this section apply to the OpenAI connector.

  • The connector ingests a fixed set of 20 tables from the OpenAI Organization Admin API. It doesn't make available other OpenAI resources, such as fine-tuning jobs, batches, files, or model inference data.
  • The costs table and the project subresource tables (project_users, project_service_accounts, project_api_keys, project_rate_limits, and project_certificates) are full refresh only. The connector re-snapshots them on each pipeline update.
  • The project subresource tables cover active projects only. The connector excludes archived projects from these tables. The standalone projects table includes archived projects.
  • The incremental tables (audit_logs and the usage_* tables) and the costs table ingest a rolling one-year history on the first run. The connector doesn't ingest usage, cost, or audit data older than one year.
  • SCD type 2 applies to the mutable entity tables (users, projects, invites, admin_api_keys, certificates, and the project_* subresource tables), whose records change over time. It doesn't apply to the append-only audit_logs, usage_*, and costs tables, which record immutable events and daily buckets.
On this page