Skip to main content
Last updated on

Microsoft Dynamics 365 connector FAQs

Preview

This feature is in Public Preview.

This page answers frequently asked questions about the Microsoft Dynamics 365 connector for Lakeflow Connect.

For general questions about managed ingestion, see Managed connector FAQs.

Connector-specific FAQs​

What Dynamics 365 applications are supported?​

The Dynamics 365 connector supports:

Dataverse-native applications (direct access, no virtual entities or direct tables required). Examples include:

  • Dynamics 365 Sales
  • Dynamics 365 Customer Service
  • Dynamics 365 Marketing
  • Dynamics 365 Field Service

Non-Dataverse-native applications (requires either virtual entities or direct tables). Examples include:

  • Dynamics 365 Finance & Operations (F&O)

See Configure data source for Microsoft Dynamics 365 ingestion for setup details.

How does the connector access D365 data?​

The Dynamics 365 connector uses Azure Synapse Link for Dataverse as an intermediary:

  1. Azure Synapse Link continuously exports D365 data to ADLS Gen2 in CSV format.
  2. Synapse Link maintains changelogs with VersionNumber timestamps for change tracking.
  3. Databricks reads the exported files from ADLS Gen2 using Microsoft Entra ID authentication.
  4. The connector processes changelogs to perform incremental ingestion.

With this architecture, you can ingest D365 data without making OData-based API calls to D365, which reduces load on your D365 environment.

What's the difference between Dataverse-native and Non-Dataverse-native apps?​

Dataverse-native apps store data directly in Dataverse tables. The connector can access these tables immediately after configuring Azure Synapse Link.

Non-Dataverse-native apps (such as F&O) store data in their own database, not in Dataverse. To ingest data from these applications, you can use either virtual entities or direct tables.

Definitions:

  • A virtual table (or entity) appears and behaves like a regular table within Dataverse, but it doesn't store any data itself. Instead, it retrieves data on-demand, directly from an external data source (for example, F&O). This means the data is not materialized within the Dataverse database, allowing users to interact with the data without duplicating or storing it.
  • A direct table is a physical copy of application data that is exported and materialized outside Dataverse (for example, in Azure Data Lake Storage via Azure Synapse Link). The data is replicated from the source system (for example, F&O) and stored as raw transactional tables that closely match the source schema. Because the data is persisted, it supports scalable analytics and historical analysis without querying the source system.

How they are used (using F&O as an example):

  • If using virtual entities, data is exposed in Dataverse through the F&O Virtual Entity solution. These virtual entities appear as read-through tables (for example, mserp_) in Dataverse and can be exported to ADLS using Azure Synapse Link. From there, Lakeflow Connect pipelines ingest the exported folders into your target tables. This option typically provides business-friendly, pre-joined views with less downstream transformation effort, since F&O entities often aggregate data from multiple underlying tables into a denormalized view.
  • If using direct tables, Azure Synapse Link enables you to export raw F&O tables directly. These tables are available in a separate section during Synapse Link setup and are written to ADLS as raw transactional data.

Considerations for deciding when to use which:

To decide which option is best, you might consider:

  • Data granularity and control: Direct tables provide the most granular, raw transactional data for ultimate control over data modeling. Virtual tables offer a more flattened, aggregated view—potentially including computed fields.
  • Data readiness and transformation effort: Relatedly, virtual tables often offer pre-joined, business-ready data, which can help to minimize downstream transformations in Databricks. Direct tables often require additional data engineering effort for complex joins and transformations.
  • Performance: While both options are reasonably efficient, they have overhead at different parts of the flow. Virtual tables can introduce overhead on the sources side (with the magnitude depending on the complexity of the entity), while direct tables may require more downstream compute for business logic.

Azure Synapse Link for Dataverse is required because:

  • Change tracking: Synapse Link provides changelogs with VersionNumber fields that enable incremental ingestion.
  • Performance: Reading exported files from ADLS Gen2 is more efficient than making OData-based API calls to D365.

How does incremental ingestion work?​

The Dynamics 365 connector uses the VersionNumber field from Azure Synapse Link changelogs to track changes:

  1. Synapse Link exports data to timestamped folders in ADLS Gen2.
  2. Each export includes a changelog file with VersionNumber values indicating when records changed.
  3. The connector processes folders in chronological order based on timestamps.
  4. For each folder, the connector reads the changelog and applies changes (inserts, updates, deletes).
  5. The connector stores the last processed VersionNumber as a cursor.
  6. Subsequent pipeline runs only process new folders created after the last cursor.

This approach ensures the connector captures all changes without reprocessing unchanged data.

See SCD types for information about how the connector handles updates and deletes.

If Azure Synapse Link stops exporting data:

  • The connector continues running but doesn't find new folders to process.
  • No new changes are ingested until Synapse Link resumes.
  • When Synapse Link resumes, it exports all missed changes to new folders.
  • The connector processes these folders during the next pipeline run.
important

If Synapse Link stops for an extended period, you might need to perform a full refresh to ensure data consistency, especially if Synapse Link's retention period expires.

Monitor your Synapse Link connection in the Power Apps maker portal to ensure continuous operation.

Can I ingest attachments from D365?​

The Dynamics 365 connector ingests attachment metadata (file name, size, MIME type, record associations) but doesn't download attachment file contents. This is because Synapse Link exports table data, not binary file contents.

To access attachment files:

  1. Ingest attachment metadata tables (for example, annotation, attachment).
  2. Use the metadata to identify files you need.
  3. Download files directly from D365 using the Dynamics 365 Web API or Power Automate.
  4. Store files in your preferred storage location (for example, ADLS Gen2, volume).

Do I need separate connections for different D365 apps?​

No, you can use a single Unity Catalog connection for all D365 applications in the same Dataverse environment. The connection authenticates to your ADLS Gen2 storage account, not to individual D365 applications.

However, you need separate pipelines for each Dataverse environment (source_schema). For example:

  • Single connection: Authenticates to your ADLS Gen2 container.
  • Multiple pipelines: One pipeline per Dataverse environment, each specifying a different source_schema value.

This approach simplifies authentication management while allowing you to ingest from multiple environments.

What permissions are required in D365?​

To set up the Dynamics 365 connector, you need:

In Dynamics 365 and Dataverse:

  • System Administrator role or equivalent permissions to configure Azure Synapse Link.
  • Read permissions for all tables you want to ingest.
  • Permissions to configure virtual entities or direct tables (for applications like F&O).

In Azure:

  • Permissions to create and configure ADLS Gen2 storage accounts and containers.
  • Permissions to create and configure Microsoft Entra ID applications.
  • Permissions to assign the Storage Blob Data Contributor role to the Entra ID application.

In Databricks:

  • Workspace administrator or metastore administrator permissions to create Unity Catalog connections.
  • CREATE permissions on the target catalog and schema.

See Configure data source for Microsoft Dynamics 365 ingestion for detailed permission requirements.

Can I ingest data from multiple Dataverse environments?​

Yes, you can ingest from multiple Dataverse environments using a single connection. Create separate pipelines for each environment:

Python
# Pipeline for production environment
prod_pipeline = w.pipelines.create(
    name="d365_prod_ingestion",
    ingestion_definition=IngestionPipelineDefinition(
        channel="PREVIEW",
        connection_name="d365_connection",  # Same connection
        source_schema="https://prod.crm.dynamics.com",  # Production
        source_table=["account", "contact"],
        destination_catalog="main",
        destination_schema="d365_prod",
        scd_type="SCD_TYPE_2"
    )
)

# Pipeline for test environment
test_pipeline = w.pipelines.create(
    name="d365_test_ingestion",
    ingestion_definition=IngestionPipelineDefinition(
        channel="PREVIEW",
        connection_name="d365_connection",  # Same connection
        source_schema="https://test.crm.dynamics.com",  # Test
        source_table=["account", "contact"],
        destination_catalog="main",
        destination_schema="d365_test",
        scd_type="SCD_TYPE_2"
    )
)

You must either have all environments export to the same ADLS Gen2 storage account and container, or create separate connections for each storage location.

How can I reduce ingestion costs?​

To optimize costs:

  • Select specific columns: Use column selection to ingest only required columns.
  • Filter tables: Only include tables you need in the pipeline.
  • Optimize history tracking: Turn history tracking off (for example, SCD type 1) if you don't need historical tracking (reduces storage).

See Microsoft Dynamics 365 ingestion connector limitations for more considerations.

Can I transform data during ingestion?​

Lakeflow Connect ingests raw data from D365 without transformations. To transform data:

  1. Ingest raw data into a landing schema (for example, d365_landing).
  2. Create downstream Lakeflow Spark Declarative Pipelines pipelines for transformations.
  3. Use SQL or Python to transform data into curated schemas.

This separation of concerns preserves raw data while allowing flexible transformations downstream.

How do I handle schema changes in D365?​

At this time, all schema changes require a full refresh of the table. Monitor your D365 schema changes and plan full refreshes accordingly.

No.