Skip to main content
Last updated on

Microsoft 365 Unified Audit Logs connector limitations

Beta

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow Connect for Microsoft 365 Unified Audit Logs from the Previews page. See Manage Databricks previews.

The managed Microsoft 365 Unified Audit Logs connector can request content created during the previous seven days. Run ingestion more frequently than every seven days to avoid gaps.

General software as a service (SaaS) connector limitations​

The limitations in this section apply to all SaaS connectors in Lakeflow Connect.

  • When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
  • When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
  • During source maintenance periods, Databricks might not be able to access your data.
  • If a source table name conflicts with an existing destination table name, the pipeline update fails.
  • Multi-destination pipeline support is API-only.
  • You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
  • If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
  • Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
  • The source system assumes that the cursor columns are monotonically increasing.
  • The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Connector-specific limitations​

The following limitations apply to the Microsoft 365 Unified Audit Logs connector:

  • The first pipeline update can ingest content from at most the previous seven days. If a pipeline doesn't run for more than seven days, the next update resumes from seven days before the current time.
  • Run each pipeline more frequently than every seven days. Microsoft does not make older Management Activity API content available through the connector.
  • Microsoft 365 makes audit content available only after its corresponding Management Activity API subscription starts. Content from before the subscription started might not be available.
  • The first content blobs can take up to 12 hours to appear after a subscription starts.
  • Don't stop or disable a Management Activity API subscription while its source table is being ingested.
  • SCD Type 2 is not supported because unified audit events are append-only.
  • Row filtering is not supported.
On this page