Microsoft Dynamics 365 connector reference
This reference covers authentication, cursor and schema behavior, supported Dataverse data types, schema evolution, and pipeline parameters for the Microsoft Dynamics 365 connector in Lakeflow Connect.
Authentication parameters
The Dynamics 365 connector uses Microsoft Entra ID (formerly Azure Active Directory) OAuth authentication. For details, see How does the connector access D365 data?.
Required authentication fields
When you create a Unity Catalog connection for D365, provide these parameters:
Parameter | Description | Example |
|---|---|---|
| Your Microsoft Entra ID tenant ID (Directory ID) |
|
| The application (client) ID of your Entra ID app |
|
| The client secret value created for your Entra ID app |
|
| The name of your ADLS Gen2 storage account |
|
| The container where Synapse Link exports data |
|
| The OAuth scope for Azure Storage access |
|
Cursor field
The Dynamics 365 connector uses the versionnumber field from Azure Synapse Link changelogs as the cursor for incremental ingestion.
Cursor behavior
- Source: Synapse Link automatically generates
versionnumbervalues when exporting changes. - Format: Integer timestamp representing the change sequence.
- Scope: Per-table cursor. Each table maintains its own cursor position.
- Storage: Cursors are stored in the pipeline metadata and don't appear in target Delta tables.
Cursor requirements
For incremental ingestion to work:
- Synapse Link must export changelogs with the
versionnumberfield. versionnumbermust be present in all changelog files.- Changelog folders must follow Synapse Link's timestamp-based naming convention.
If versionnumber is missing, incremental ingestion fails and you must perform a full refresh.
Source export format
Azure Synapse Link can export Dataverse data to ADLS Gen2 in either CSV or Parquet format. The Dynamics 365 connector supports both and automatically detects which format Synapse Link wrote, so you don't specify the export format in the pipeline definition. The format is determined by your Synapse Link setup. Exporting as Parquet requires connecting an Azure Synapse Analytics workspace (see Configure a Parquet data source for Microsoft Dynamics 365 ingestion), while CSV export uses the standard setup (see Configure data source for Microsoft Dynamics 365 ingestion).
- CSV: Synapse Link writes CSV files directly to ADLS Gen2 without an Azure Synapse Analytics workspace.
- Parquet: Parquet ingestion is in Beta. Synapse Link writes each table as a Parquet-format Delta table under
<profileRoot>/deltalake/<tableName>/. This path requires an Azure Synapse Analytics workspace and an Apache Spark pool, and Databricks recommends it for large or high-volume instances.
Schema discovery
The Dynamics 365 connector automatically discovers table schemas from Dataverse metadata.
Discovery process
When you create a pipeline:
- The connector reads Synapse Link metadata files from ADLS Gen2.
- The connector extracts table schemas from the metadata JSON files.
- Column names, data types, and nullability are inferred from the metadata.
- Target tables are created with the discovered schemas.
Schema evolution for CSV ingestion
This feature is in Private Preview. To try it, reach out to your Databricks contact.
Schema evolution automatically keeps your target Delta tables in sync as the source Dataverse schema changes, including column additions, deletions, renames from Azure Synapse Link.
Prerequisites
- Schema evolution is opt-in during Private Preview. To enable it for your pipeline, reach out to your Databricks contact.
- The connection's service principal must have the Storage Blob Data Contributor role on the ADLS Gen2 storage account. Schema evolution writes the schema checkpoint back to your storage, so any role that grants only read permission does not work. A read-only connection fails with an error asking you to grant write access or disable schema evolution. For connection setup, see Create a Dynamics 365 connection.
How schema changes are applied
A pipeline update reads with a single, fixed schema for its whole duration, so the connector starts a new update to pick up a changed schema. Each schema change in the source is applied as follows:
- The update ingests the exported data up to, but not including, the point where the schema changes, and commits it.
- The update cancels, and the connector records the new schema.
- A new update starts at the point where the schema changed and continues from there with the new schema.
Each schema change in the export history repeats this cycle, so a backlog that spans several schema changes takes several updates to work through. The connector restarts the updates for you, no action is needed.
The restart is designed so that data is neither duplicated nor lost. Each update commits only the data it read before the change, and the next update resumes at the point where the previous one stopped.
Because these restarts are expected, a cancelled update following a source schema change is not a failure. The pipeline reports a schema change event that names the affected table, along with any new columns.
Supported and unsupported changes
Schema evolution handles the following source schema changes:
Change | Supported |
|---|---|
Column addition | Supported |
Column deletion | Supported |
Column rename | Supported |
Table addition | Supported |
Table removal | Supported |
Removing a column and later re-adding a column with the same name | Not supported |
Removing a table and later re-adding a table with the same name | Not supported |
Supported Dataverse data types
The Dynamics 365 connector maps Dataverse data types to Delta Lake data types.
Data type mapping
Dataverse type | Delta Lake type | Notes |
|---|---|---|
|
| Max length preserved as metadata |
|
| |
|
| |
|
| |
|
| Precision and scale preserved |
|
| |
|
| Stored as decimal with 4 decimal places |
|
| |
|
| Timezone information preserved |
|
| |
|
| Spark has no native |
|
| Stored as string representation |
|
| Foreign key GUID stored as string |
|
| Integer value, not label |
|
| Comma-separated integer values |
|
| URL or metadata, not binary data |
|
| Metadata only, not file contents |
Complex data types
Some Dataverse types require special handling:
Dataverse type | Ingested as | How to handle it |
|---|---|---|
| Integer codes | Join with the |
| GUID strings | Join with the referenced table to get related data |
| Comma-separated integer strings, such as | Parse the string to extract individual values |
The following examples parse a multi-select option set, first splitting the comma-separated values into an array and then exploding them into separate rows:
-- Split comma-separated values into array
SELECT
accountid,
accountname,
SPLIT(industrycodes, ',') AS industry_array
FROM main.d365_data.account;
-- Explode into separate rows
SELECT
accountid,
accountname,
CAST(code AS INT) AS industry_code
FROM main.d365_data.account
LATERAL VIEW EXPLODE(SPLIT(industrycodes, ',')) AS code;
API version compatibility
The Dynamics 365 connector is compatible with:
- Dataverse API: Version 9.2 and later
- Azure Synapse Link for Dataverse: Version 1.0 and later
- Azure Storage REST API: Version 2021-08-06 and later
- Microsoft Entra ID: OAuth 2.0 client credentials flow
Older API versions might work but aren't officially supported. Keep your D365 and Azure services updated for best compatibility.
Incremental ingestion behavior
How the connector applies each detected change depends on the SCD type you choose for the pipeline, and whether deletes reach Databricks at all depends on your Synapse Link configuration.
Change detection
The connector detects every change from the Synapse Link changelogs. It treats a record's presence in a changelog as an insert, a changed versionnumber as an update, and a delete marker as a delete. Delete markers appear only if Synapse Link is configured to export deletes.
SCD Type 1 behavior
For SCD Type 1 pipelines, records are updated in place without preserving history. Updates overwrite existing rows based on primary key, and deletes remove rows (if delete tracking is enabled).
Querying a target table returns one row per record, reflecting only its latest state:
SELECT * FROM main.d365_data.account ORDER BY accountid;
-- Result: Latest state only
-- accountid | accountname | modifiedon
-- 123 | Acme Corp | 2025-12-03 10:00:00
-- 456 | TechCo | 2025-12-03 09:30:00
SCD Type 2 behavior
For SCD Type 2 pipelines, all changes are preserved as new row versions. The connector adds __START_AT, __END_AT, and __CURRENT columns to track version history.
Querying a target table returns every version of each record. __START_AT and __END_AT bound the window in which each version was current, and the active version has a NULL __END_AT and __CURRENT set to true:
SELECT * FROM main.d365_data.account ORDER BY accountid, __START_AT;
-- Result: All historical versions
-- accountid | accountname | __START_AT | __END_AT | __CURRENT
-- 123 | Acme Inc | 2025-11-01 08:00:00 | 2025-12-03 10:00:00 | false
-- 123 | Acme Corp | 2025-12-03 10:00:00 | NULL | true
-- 456 | TechCo | 2025-12-01 14:00:00 | NULL | true
When Synapse Link exports as Parquet, older Delta checkpoints are periodically compacted into newer ones, which can make some historical record versions unavailable for processing. For SCD Type 2 pipelines, this can produce incomplete history, but it doesn't cause data loss: the pipeline always reflects the latest snapshot of each record correctly, and only some intermediate versions might be missing. To reduce the chance of incomplete history, Databricks recommends running the pipeline more frequently than once every 24 hours, which narrows the window in which compaction can drop historical changes.
Delete handling
Delete handling depends on your Synapse Link configuration:
- Hard deletes: If Synapse Link exports delete records, the connector removes (SCD Type 1) or marks (SCD Type 2) deleted records.
- No delete tracking: If Synapse Link doesn't export deletes, deleted records remain in target tables until you perform a full refresh.
Verify your Synapse Link configuration exports deletes if you need accurate delete tracking.
Pipeline parameters
When creating a D365 ingestion pipeline, specify these parameters:
Required parameters
These parameters must be specified for the pipeline to run.
Parameter | Type | Description | Example |
|---|---|---|---|
| String | Must be |
|
| String | Name of your Unity Catalog connection |
|
| String | Synapse Link logical schema name, typically |
|
| String | D365 table logical name, with one entry per |
|
| String | Target Unity Catalog catalog |
|
| String | Target Unity Catalog schema |
|
| String |
|
|
Optional parameters
These parameters can optionally be set when creating your pipeline.
Parameter | Type | Description | Example |
|---|---|---|---|
| Object | Per-table settings, such as column selection | See column selection |
Example pipeline configuration
This is an example of a complete pipeline configuration using Python SDK.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.pipelines import IngestionPipelineDefinition
w = WorkspaceClient()
pipeline = w.pipelines.create(
name="d365_comprehensive_ingestion",
ingestion_definition=IngestionPipelineDefinition(
channel="PREVIEW",
connection_name="d365_connection",
source_schema="objects",
source_table="account",
destination_catalog="main",
destination_schema="d365_sales",
scd_type="SCD_TYPE_2",
table_configuration={
"account": {
"columns": [
"accountid",
"accountnumber",
"name",
"emailaddress1",
"telephone1"
]
}
}
)
)
Finding table logical names
To identify table logical names for the source_table parameter:
- Power Apps maker portal: Navigate to Tables and view the Logical name column.
- Dataverse API: Query metadata using
https://yourorg.api.crm.dynamics.com/api/data/v9.2/EntityDefinitions. - ADLS Gen2 storage: List folders in your Synapse Link container (folder names match logical names).
Use lowercase logical names in pipeline configurations (for example, "account" rather than "Account"). The connector is case-sensitive.
Performance tuning
The connector offers limited tuning, because Synapse Link controls the export itself. What you can control is how much data each pipeline moves and how you distribute tables across pipelines.
Column selection
Selecting only the columns you need reduces data transfer from ADLS Gen2, storage costs in Delta Lake, and query processing time. See column selection for configuration details.
Table grouping
Group tables that behave alike, so a pipeline's schedule suits everything in it: related tables together for easier management, and tables with similar update patterns together so you can schedule each pipeline to match its change volume. Give high-volume tables their own pipelines, so a single large table doesn't slow the rest.
Each pipeline is limited to 250 tables. For larger environments, create multiple pipelines.
Troubleshooting
For common issues and solutions when working with the Dynamics 365 connector, see Troubleshoot Microsoft Dynamics 365 ingestion.