Skip to main content
Last updated on

Configure a Parquet data source for Microsoft Dynamics 365 ingestion

Beta

Parquet ingestion is in Beta.

Learn how to set up Microsoft Dynamics 365 as a data source for ingestion into Databricks using Lakeflow Connect when your data is exported as Parquet. This workflow uses an Azure Synapse Analytics workspace and an Apache Spark pool to export Dataverse tables as Parquet-format Delta tables, and then configures the authentication that Databricks uses to read them.

note

This page covers the Parquet workflow, which requires an Azure Synapse Analytics workspace and an Apache Spark pool. Parquet export uses Azure Synapse Link's Delta Lake export to ADLS Gen2, which isn't the same as Microsoft Fabric Link. The connector doesn't support Fabric Link. If your Azure Synapse Link exports data as CSV, use Configure data source for Microsoft Dynamics 365 ingestion instead.

For information about how the connector accesses your source data, see How does the connector access D365 data?. For a list of supported Dataverse applications, see Which Dynamics 365 applications are supported?.

Prerequisites​

Before you configure the Dynamics 365 data source, you must have:

  • An active Azure subscription with permissions to create resources.
  • A Microsoft Dynamics 365 environment with administrator access.
  • A Dataverse environment associated with your Dynamics 365 instance.
  • Workspace administrator or metastore administrator permissions in Databricks.
  • Permissions to create and configure Azure Synapse Link in your Dataverse environment.
  • Permissions to create an Azure Synapse Analytics workspace and an Apache Spark pool.
  • An ADLS Gen2 storage account (or permissions to create one).
  • Permissions to create and configure Microsoft Entra ID applications.
  • Dataverse API v9.2 or later.
  • Azure Storage REST API version 2021-08-06.
  • Azure Synapse Link for Dataverse version 1.0 or later.

Create an Azure Synapse Analytics workspace​

The Synapse Analytics workspace hosts the Apache Spark pool that Azure Synapse Link uses to export your Dataverse tables as Parquet.

  1. In the Azure portal, go to Azure Synapse Analytics, then click Create.
  2. Select the subscription, resource group, and Data Lake Storage Gen2 account to associate with the workspace.
  3. Select Assign myself the Storage Blob Data Contributor role on the Data Lake Storage Gen2 account to interactively query in the workspace.
  4. On the Security tab, under Authentication method, select Use only Microsoft Entra ID authentication.
  5. Click Review + Create, then click Create.
  6. Wait for the deployment to complete before continuing.

Create an Apache Spark pool​

Azure Synapse Link uses an Apache Spark pool to write the exported Parquet-format Delta tables.

  1. From the workspace Overview page, click New Apache Spark pool.
  2. Provide a name for the pool.
  3. For large datasets, increase the Node size to Large and set the Number of nodes to 10. You don't need any additional settings.
  4. Click Review + Create, then click Create.

In this step, you use Azure Synapse Link to choose the tables that you want to ingest and export them as Parquet through the Spark pool.

Configure virtual entities or direct tables (optional)​

Virtual entities and direct tables make data from non-Dataverse sources (like Dynamics 365 Finance & Operations) available in Dataverse without copying the data. For non-Dataverse sources, you must configure virtual entities or direct tables before setting up Azure Synapse Link.

To configure virtual entities:

  1. In Power Apps, go to the Environments page, then click Dynamics 365 apps.
  2. To link F&O entities as virtual entities in Dataverse, install the Finance and Operations Virtual Entity solution.
  3. Set up Service To Service (S2S) authorization between Dataverse and your F&O application. With this authorization, Dataverse can communicate with your application. For details, see the Microsoft documentation Configure Dataverse virtual entities.
  4. For every virtual entity that you wish to ingest, enable Track Changes under Advanced Properties.
  5. By default, the F&O Virtual Entity solution exposes some virtual entities in the list of Dataverse tables. However, you can expose additional entities manually:
    1. Go to your Dataverse environment's Advanced Settings page.
    2. Click the filter icon in the upper right to access the advanced search.
    3. Select Available Finance and Operation Entities from the drop-down menu, then click Results.
    4. Select the virtual entity that you want to expose.
    5. On the Entity Admin page, toggle Visible to True, then click Save and Close.

You can now see the entity in the list of Dataverse tables with a name that starts with mserp_.

important

Virtual entities and direct tables appear in Azure Synapse Link only after Dataverse finishes synchronizing them. This usually takes up to 15 minutes, but can take up to 30. If tables are missing after 30 minutes, see Virtual entities not appearing in schema discovery.

  1. Go to make.powerapps.com and click Tables.

  2. At the top of the page, click Analyze, then Azure Synapse Link.

  3. Click New Link.

  4. Select the Connect to your Azure Synapse Analytics workspace checkbox, then fill in the details. Select the Apache Spark pool and the storage account that you created in the previous steps. Selecting this checkbox is what exports your data as Parquet, and there is no separate export-format option to set.

  5. Select the table or tables that you want to sync.

    • If ingesting from a Dataverse-native app, select the relevant Dataverse tables directly from the Dataverse section.
    • If ingesting from F&O, you can either select direct tables from the D365 Finance & Operations section or virtual entities from the Dataverse section (prefix mserp_). For more information about virtual entities, see Configure virtual entities or direct tables (optional).

  6. Click Save, then wait for the initial sync to complete.

    Synapse Link writes each table as a Parquet-format Delta table under <profileRoot>/deltalake/<tableName>/ in the associated ADLS Gen2 storage account.

  7. To look at the export, click Go to Azure Synapse Analytics Workspace.

    note

    For F&O users, this initial sync can take hours for large tables with hundreds of gigabytes.

Configure authentication for Databricks​

You must set up authentication as a default, based on your Dynamics configuration. In this step, you collect the Microsoft Entra ID and storage details needed to create a Unity Catalog connection, and then grant the Microsoft Entra ID application access to the exported Parquet-format Delta tables.

Collect Entra ID and storage details​

  1. Collect the tenant ID of your Entra ID tenant (portal.azure.com >> Microsoft Entra ID >> Overview tab >> Tenant ID, listed on the right-hand pane).
  2. When you create an Azure Synapse Link, Azure Synapse creates an ADLS container in the associated storage account for the exported Parquet-format Delta tables. Locate the ADLS container name by visiting the Synapse Link's Admin Page.
  3. Collect the Azure Storage account name of the storage account associated with the workspace.
  4. Collect the access credentials for the ADLS container.
    1. Create a Microsoft Entra ID App, if you don't have one already.
    2. Collect the client secret.
    3. Collect the App ID (portal.azure.com >> Microsoft Entra ID >> Manage >> App Registrations).

Grant the Entra ID application access to storage​

Grant the Entra ID App access to the ADLS container, if you haven't already.

note

Verify that your Entra ID application has access to the ADLS containers associated with each Synapse Link profile. If you're ingesting data from multiple environments or applications, confirm that the application has role assignments on all relevant containers.

  1. Go to Azure Storage Accounts and select your container or storage account. (Databricks recommends the container level to maintain least privileges.)
  2. Click Access Control (IAM), then Add role assignment.
  3. Select the Storage Blob Data Contributor → Read/Write/Delete access role. If your organization doesn't allow this, contact your Databricks account team.
  4. Click Next, then Select Members.
  5. Select User, group, or service principal, then Search for your App Registration. (If the app is not present in the search result, you can explicitly enter its object ID in the search bar, then press Enter).
  6. Click Review + Assign.
  7. To confirm that the permissions are configured correctly, you can check your container's Access Control.

Next steps​

After you configure the Parquet data source, create a connection and an ingestion pipeline in Databricks:

On this page