Microsoft Dynamics 365 connector limitations
This page describes limitations and restrictions for the Microsoft Dynamics 365 connector in Lakeflow Connect.
General SaaS connector limitations
The Dynamics 365 connector shares limitations common to all SaaS connectors in Lakeflow Connect:
- When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
- When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
- During source maintenance periods, Databricks might not be able to access your data.
- If a source table name conflicts with an existing destination table name, the pipeline update fails.
- Multi-destination pipeline support is API-only.
- You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
- If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
- Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
- The source system assumes that the cursor columns are monotonically increasing.
- The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.
Azure Synapse Link requirements
The Dynamics 365 connector requires Azure Synapse Link for Dataverse to run continuously:
- Synapse Link must export data without interruption. If Synapse Link stops, the connector can't capture changes until it resumes.
- Changes appear in Azure Data Lake Storage (ADLS) Gen2 after Synapse Link's export interval, which is typically 5 to 15 minutes. The Synapse Link architecture inherently includes this latency.
- Configure appropriate retention policies for your ADLS Gen2 storage. If you delete Synapse Link exports before ingestion, you must perform a full refresh.
- Synapse Link can export in either CSV or Parquet format. Parquet ingestion is in Beta. The connector supports both and detects the format automatically. Parquet export additionally requires an Azure Synapse Analytics workspace and an Apache Spark pool. See Configure a Parquet data source for Microsoft Dynamics 365 ingestion.
If Azure Synapse Link stops for an extended period, you might miss changes. Monitor Synapse Link health in the Power Apps maker portal and set up alerts for export failures.
Schema evolution
Schema evolution for CSV ingestion of standard (non-virtual) entities is available in Private Preview. For supported changes and how to enable it, see Microsoft Dynamics 365 connector reference. Virtual entities are not covered.
Virtual entity schema evolution
Finance & Operations virtual entities need extra handling, because a schema change must propagate through Dataverse before the connector can detect it. When an F&O virtual entity changes:
- Refresh the virtual entity configuration in Dataverse.
- Allow up to 15 minutes for the change to appear in Dataverse schema discovery.
- Perform a full refresh of the affected tables.
Because these changes originate in F&O rather than in Databricks, monitor virtual entity updates and coordinate full refreshes with your Dynamics 365 administrator.
Incremental ingestion
The Dynamics 365 connector's incremental ingestion has these limitations:
- Synapse Link must export changelogs with the
versionnumberfield. Ifversionnumberis missing, you must use full refresh mode. - The connector processes Synapse Link export folders in chronological order. If you delete folders or folders are missing, the connector can't recover without a full refresh.
- If Synapse Link misses changes due to downtime, those changes aren't captured unless you perform a full refresh.
- The connector detects deletes only if Synapse Link exports delete records in changelogs. Some Dynamics 365 configurations don't export deletes.
- With Parquet export, Delta checkpoint compaction can make some historical record versions unavailable, so SCD Type 2 pipelines might have incomplete history (though never data loss, because the latest snapshot is always correct). Databricks recommends running pipelines more frequently than once every 24 hours to minimize this. See SCD Type 2 behavior.
Verify your Synapse Link configuration exports changelogs with versionnumber before creating pipelines. See Configure data source for Microsoft Dynamics 365 ingestion.
Attachments and files
The Dynamics 365 connector ingests attachment metadata but not file contents:
- The connector ingests attachment tables such as
annotationandattachment, including file names, sizes, MIME types, and record associations. - The connector doesn't ingest file contents. You must download files separately using the Dynamics 365 Web API or Power Automate.
- Synapse Link exports table data, not binary files stored in Microsoft Dynamics 365.
You can instead use attachment metadata to identify required files, then download them using Dynamics 365 APIs and store in a volume or ADLS Gen2.
Data type support
The Dynamics 365 connector supports most Dataverse data types with full fidelity. Complex types are ingested in a simplified form and need downstream handling.
Dataverse data type | Support | What to do downstream |
|---|---|---|
String, Integer, Decimal, Boolean, DateTime, Money | Full fidelity | Nothing. These types ingest as-is. |
Option set (picklist) | Ingested as integer values | Join with the |
Multi-select option set | Ingested as comma-separated integer strings | Parse the string to extract individual values. |
Lookup | Ingested as GUIDs | Join with the referenced table to get related record data. |
Complex JSON object | Ingested as strings | Parse the JSON in downstream transformations. Applies to custom Dataverse types that export as JSON. |
Image | Metadata only | Download image data separately. See Attachments and files. |
See Microsoft Dynamics 365 connector reference for a complete data type mapping table.
Performance considerations
The initial sync and later incremental syncs are bound by different factors, so they need different tuning:
Sync phase | What drives the time | How to reduce it |
|---|---|---|
Initial sync | Data volume, mostly. Tables with millions of records take longer, and the initial Synapse Link export alone can take hours for large datasets. Network throughput between Azure and Databricks also affects transfer speed. | Start with a small subset of tables to validate the setup, then add more tables incrementally. Network throughput isn't adjustable from the pipeline. |
Incremental sync | Change volume, such as thousands of updates per minute. The number of Synapse Link export folders and the size of each changelog both add processing overhead. | Schedule pipeline runs to match your change volume. Running more often keeps each run's changelog smaller, which offsets the cost of processing more folders. |
Limitations
A single pipeline can ingest a maximum of 250 tables. This limit is per pipeline, not per connection, so to ingest more than 250 tables, create multiple pipelines using the same connection.
The connector also depends on minimum versions of the services it reads from:
- Dataverse API v9.2 or later. Older versions might not be supported.
- Azure Storage REST API version 2021-08-06. Make sure that your storage account supports this version.
- Azure Synapse Link for Dataverse version 1.0 or later.
Keep your Dynamics 365, Dataverse, and Azure services updated to the latest versions for best compatibility.
Known issues
- Virtual entities sometimes take longer than 15 minutes to synchronize. If tables don't appear, wait up to 30 minutes and retry.
- Synapse Link occasionally fails to export specific tables. Check Synapse Link logs in Power Apps for errors.
- In rare cases, changelogs might have gaps in
versionnumbersequences. This doesn't affect data integrity but might cause warnings in pipeline logs.
Report issues to Databricks support with pipeline IDs and timestamps for investigation.