Microsoft Dynamics 365 connector FAQs

Find answers to frequently asked questions about the managed Microsoft Dynamics 365 connector in Lakeflow Connect. For general questions about managed ingestion connectors, see Managed connector FAQs.

Connector-specific FAQs

The following questions cover how the connector reads your data through Azure Synapse Link, which Dynamics 365 applications and permissions it requires, and how to plan connections, costs, and schema changes.

How does the connector access D365 data?

The Dynamics 365 connector uses Azure Synapse Link for Dataverse as an intermediary:

1. Synapse Link continuously exports D365 data to ADLS Gen2 in CSV format.
2. Synapse Link maintains changelogs with VersionNumber timestamps for change tracking.
3. Databricks reads the exported files from ADLS Gen2 using Microsoft Entra ID authentication.
4. The connector processes changelogs to perform incremental ingestion.

With this architecture, you can ingest D365 data without making OData-based API calls to D365, which reduces load on your D365 environment.

Why do I need Azure Synapse Link?

Azure Synapse Link for Dataverse is required for:

• Change tracking: Synapse Link provides changelogs with versionnumber fields that enable incremental ingestion.
• Performance: Reading exported files from ADLS Gen2 is more efficient than making OData-based API calls to D365.

How does incremental ingestion work?

The Dynamics 365 connector uses the versionnumber field from Azure Synapse Link changelogs to track changes:

1. Synapse Link exports data to timestamped folders in ADLS Gen2.
2. Each export includes a changelog file with versionnumber values indicating when records changed.
3. The connector processes folders in chronological order based on timestamps.
4. For each folder, the connector reads the changelog and applies changes (inserts, updates, deletes).
5. The connector stores the last processed versionnumber as a cursor.
6. Subsequent pipeline runs only process new folders created after the last cursor.

This approach ensures the connector captures all changes without reprocessing unchanged data. See Enable history tracking (SCD type 2) for information about how the connector handles updates and deletes.

Which Dynamics 365 applications are supported?

The Dynamics 365 connector supports Dataverse-native applications and non-Dataverse-native applications.

Dataverse-native applications, which the connector accesses directly without virtual entities or direct tables, include:

• Dynamics 365 Sales
• Dynamics 365 Customer Service
• Dynamics 365 Marketing
• Dynamics 365 Field Service

Non-Dataverse-native applications, which require either virtual entities or direct tables, include:

• Dynamics 365 Finance & Operations (F&O)

See Configure data source for Microsoft Dynamics 365 ingestion for setup details.

What's the difference between Dataverse-native and non-Dataverse-native apps?

Dataverse-native apps store data directly in Dataverse tables. The connector can access these tables immediately after you configure Azure Synapse Link.

Non-Dataverse-native apps such as F&O store data in their own database rather than in Dataverse. To ingest their data, you use either virtual entities or direct tables:

• A virtual table, also called a virtual entity, appears and behaves like a regular Dataverse table but stores no data itself. It retrieves data on demand from the external source, so the data is never materialized in Dataverse. You can work with the data without duplicating it.
• A direct table is a physical copy of application data, exported and materialized outside Dataverse. Azure Synapse Link replicates the data from the source system into ADLS Gen2 as raw transactional tables that closely match the source schema. Because the data persists, it supports scalable analytics and historical analysis without querying the source system.

Taking F&O as an example:

• Virtual entities are exposed in Dataverse through the F&O Virtual Entity solution. They appear as read-through tables prefixed with mserp_, which Azure Synapse Link exports to ADLS Gen2 for Lakeflow Connect pipelines to ingest. F&O entities often aggregate several underlying tables into a denormalized view, so this option usually gives you business-friendly, pre-joined data that needs less downstream transformation.
• Direct tables are raw F&O tables that Azure Synapse Link exports without routing them through Dataverse. They appear in a separate section during Synapse Link setup and land in ADLS Gen2 as raw transactional data.

To decide which option fits your workload:

Consideration	Virtual tables	Direct tables
Data granularity	A flatter, aggregated view that can include computed fields.	The most granular raw transactional data, giving you full control over data modeling.
Transformation effort	Often pre-joined and business-ready, which minimizes downstream transformations in Databricks.	Usually require additional data engineering for complex joins and transformations.
Performance	Can add overhead on the source side, depending on the entity's complexity.	Can require more downstream compute to apply business logic.

Consideration	Virtual tables	Direct tables
Data granularity	A flatter, aggregated view that can include computed fields.	The most granular raw transactional data, giving you full control over data modeling.
Transformation effort	Often pre-joined and business-ready, which minimizes downstream transformations in Databricks.	Usually require additional data engineering for complex joins and transformations.
Performance	Can add overhead on the source side, depending on the entity's complexity.	Can require more downstream compute to apply business logic.

Can I ingest attachments from D365?

The Dynamics 365 connector ingests attachment metadata (file name, size, MIME type, record associations) but doesn't download attachment file contents. This is because Synapse Link exports table data, not binary file contents.

To access attachment files:

1. Ingest attachment metadata tables such as annotation and attachment.
2. Use the metadata to identify files you need.
3. Download files directly from D365 using the Dynamics 365 Web API or Power Automate.
4. Store the files in your preferred storage location, such as ADLS Gen2 or a Unity Catalog volume.

Do I need separate connections for different D365 apps?

No, you can use a single Unity Catalog connection for all D365 applications in the same Dataverse environment. The connection authenticates to your ADLS Gen2 storage account, not to individual D365 applications.

However, you need separate pipelines for each Dataverse environment, identified by its source_schema value. For example:

• Single connection: Authenticates to your ADLS Gen2 container.
• Multiple pipelines: One pipeline per Dataverse environment, each specifying a different source_schema value.

This approach simplifies authentication management while allowing you to ingest from multiple environments.

What permissions are required in D365?

Setting up the Dynamics 365 connector requires permissions in three places: Microsoft Dynamics 365 and Dataverse, Azure, and Databricks.

In Microsoft Dynamics 365 and Dataverse, you need the following permissions:

• System Administrator role or equivalent permissions to configure Azure Synapse Link.
• Read permissions for all tables you want to ingest.
• Permissions to configure virtual entities or direct tables (for applications like F&O).

In Azure, you need the following permissions:

• Permissions to create and configure ADLS Gen2 storage accounts and containers.
• Permissions to create and configure Microsoft Entra ID applications.
• Permissions to assign the Storage Blob Data Contributor role to the Entra ID application.

In Databricks, you need the following permissions:

• Workspace administrator or metastore administrator permissions to create Unity Catalog connections.
• CREATE permissions on the target catalog and schema.

See Configure data source for Microsoft Dynamics 365 ingestion for detailed permission requirements.

Can I ingest data from multiple Dataverse environments?

Yes, you can ingest from multiple Dataverse environments using a single connection. Create separate pipelines for each environment:

Python

# Pipeline for production environment
prod_pipeline = w.pipelines.create(
    name="d365_prod_ingestion",
    ingestion_definition=IngestionPipelineDefinition(
        channel="PREVIEW",
        connection_name="d365_connection",  # Same connection
        source_schema="https://prod.crm.dynamics.com",  # Production
        source_table=["account", "contact"],
        destination_catalog="main",
        destination_schema="d365_prod",
        scd_type="SCD_TYPE_2"
    )
)

# Pipeline for test environment
test_pipeline = w.pipelines.create(
    name="d365_test_ingestion",
    ingestion_definition=IngestionPipelineDefinition(
        channel="PREVIEW",
        connection_name="d365_connection",  # Same connection
        source_schema="https://test.crm.dynamics.com",  # Test
        source_table=["account", "contact"],
        destination_catalog="main",
        destination_schema="d365_test",
        scd_type="SCD_TYPE_2"
    )
)

You must either have all environments export to the same ADLS Gen2 storage account and container, or create separate connections for each storage location.

How can I reduce ingestion costs?

To optimize costs:

• Use column selection to ingest only required columns. See Select columns to ingest.
• Only include tables you need in the pipeline.
• Turn history tracking off (SCD type 1) if you don't need historical tracking to reduce storage.

See Microsoft Dynamics 365 connector limitations for more considerations.

Can I transform data during ingestion?

Lakeflow Connect ingests raw data from Microsoft Dynamics 365 without transformations. To transform data:

1. Ingest raw data into a landing schema such as d365_landing.
2. Create downstream Lakeflow pipelines for transformations.
3. Use SQL or Python to transform data into curated schemas.

This separation of concerns preserves raw data while allowing flexible transformations downstream.

How do I handle schema changes in D365?

At this time, all schema changes require a full refresh of the table. Monitor your D365 schema changes and plan full refreshes accordingly.

Does the Dataverse Synapse Link instance need to be in the same region as the Databricks workspace?

No.