Troubleshoot Microsoft Dynamics 365 ingestion

This page provides troubleshooting guidance for common issues with the Microsoft Dynamics 365 connector in Lakeflow Connect. For general troubleshooting guidance that applies to all managed ingestion pipelines, see Troubleshoot managed ingestion pipelines.

Because the connector reads what Azure Synapse Link exports to ADLS Gen2, most ingestion failures start with the export rather than the pipeline. Confirm that Synapse Link is running and writing files before you investigate the pipeline itself.

Synapse Link is not exporting data

Symptoms: No folders appear in your ADLS Gen2 container after configuring Synapse Link, folder timestamps stop updating, or pipeline runs fail with "No data found" errors.

Cause: The Synapse Link connection is paused or stopped, the Azure storage account permissions are incorrect, the selected tables aren't configured for export, or Synapse Link encountered an error during export.

Resolution:

1. Check the Synapse Link status:
   • Sign in to Power Apps.
   • Go to Azure Synapse Link in your environment.
   • Verify your connection shows "Active" status.
   • If paused, click Resume to restart the export.
2. Verify storage permissions:
   • In the Azure portal, go to your storage account.
   • Click Access Control (IAM).
   • Verify the Synapse Link managed identity has the Storage Blob Data Contributor role.
   • If the role is missing, add the role assignment.
3. Check the table configuration:
   • In Power Apps, select your Synapse Link connection.
   • Review the list of selected tables and verify the tables you want to ingest are included.
   • Add missing tables and wait 5 to 15 minutes for the initial export.
4. Review the Synapse Link logs:
   • In Power Apps, select your Synapse Link connection.
   • Click View logs or History.
   • Look for error messages indicating export failures, and address the specific errors (for example, storage quota or permissions).

Synapse Link is active, but files are not appearing

Symptoms: Synapse Link shows "Active" status, but no files appear in your ADLS Gen2 container, or pipeline runs fail with "No data found" errors.

Cause: Synapse Link hasn't completed its initial export, the Synapse Link profile is paused or encountered an export error, or the storage account permissions are incorrect.

Resolution:

The connector supports both CSV and Parquet export and automatically detects which format Synapse Link writes, so you don't need to reconfigure the export format if files are missing. Parquet ingestion is in Beta. To troubleshoot, confirm that Synapse Link is actually exporting data:

1. Verify that files exist in ADLS Gen2:
   • In the Azure portal, go to your ADLS Gen2 storage account and container.
   • Open a table folder and confirm it contains data files. For CSV export, files have a .csv extension. For Parquet export, Synapse Link writes each table as a Parquet-format Delta table under <profileRoot>/deltalake/<tableName>/.
   • If the folders are empty, the initial export might still be in progress.
2. Check the Synapse Link status:
   • In Power Apps, open your Synapse Link profile and confirm it shows "Active" status.
   • If paused or stopped, click Resume to restart the export.
   • Review the Synapse Link logs or history for export errors, and address any that appear (for example, storage quota or permissions).
3. Wait for the export to complete:
   • The initial Synapse Link export can take hours for large datasets.
   • After files appear in your container, retry your pipeline run.

Error: FILE_PATH_DOES_NOT_EXIST

Symptoms: Pipeline runs fail with a FILE_PATH_DOES_NOT_EXIST error, the connector can't find expected files in ADLS Gen2, or the error indicates missing folders or file paths.

Cause: The Enable Incremental Update Folder Structure option wasn't turned on when Synapse Link was configured, so the connector doesn't find files where it expects them.

Resolution:

1. Turn on the incremental update folder structure:
   • In Power Apps, edit your Synapse Link connection.
   • Click Advanced to show advanced configuration settings.
   • Turn on Enable Incremental Update Folder Structure.
   • Save the configuration.
   • Wait for Synapse Link to regenerate the folder structure. This can take several hours for large datasets.
2. Verify the folder structure:
   • In the Azure portal, go to your ADLS Gen2 storage account and container.
   • Verify that table folders now contain timestamped subfolders (for example, 2025-12-19T10-30-00-000Z). These timestamped folders contain the incremental updates the connector needs.
3. Retry the pipeline. The connector now finds files in the expected locations.

Synapse Link changelog is missing versionnumber

Symptoms: Pipeline runs fail with "versionnumber field not found" errors, incremental ingestion doesn't work, or only full refresh succeeds.

Cause: Synapse Link isn't configured to export changelogs, change tracking isn't turned on for your tables, or your Synapse Link version is outdated.

Resolution:

1. Turn on change tracking:
   • In Power Apps, edit your Synapse Link connection.
   • Verify that Enable change tracking is selected.
   • Save and wait up to 30 minutes for Synapse Link to regenerate exports.
2. Verify the changelog files:
   • In the Azure portal, go to your ADLS Gen2 container.
   • Open a table folder and locate the SynapseLink subfolder.
   • Open a recent changelog file (CSV or JSON) and verify it contains a versionnumber column.
   • If the column is missing, contact Microsoft support to turn on change tracking.
3. Update Synapse Link. Verify you're using Azure Synapse Link for Dataverse version 1.0 or later, because older versions might not support versionnumber.
4. Perform a full refresh. If change tracking can't be turned on, you can only use full refresh mode. Full refresh reloads all data on every run, which is slower and more expensive.

Error: The selected storage account has restricted network access

Symptoms: Synapse Link setup fails with the following error:

The selected storage account has restricted network access. To proceed, please setup an enterprise policy and connect it to your Dataverse environment. Once done, please enable the 'Select Enterprise Policy with Managed Service Identity' option below.

Cause: Your ADLS staging location is secured by a firewall, and Dataverse can't reach it.

Resolution: Set up a managed identity (formerly managed service identity) to access your data. See Use managed identities for Azure with your Azure data lake storage in the Microsoft documentation.

Microsoft Entra ID authentication fails

Symptoms: Pipeline creation fails with "Authentication failed" errors, the connection test fails in Catalog Explorer, or pipeline runs fail with "401 Unauthorized" errors.

Cause: The tenant ID, client ID, or client secret is incorrect, the client secret expired, the application lacks the required permissions, or the scope is incorrect.

Resolution:

1. Verify the authentication parameters:
   • In the Azure portal, go to Microsoft Entra ID > App registrations.
   • Locate your application and verify that the Application (client) ID and Directory (tenant) ID match your connection configuration.
   • Copy the correct values and update your connection if needed.
2. Check the client secret expiration:
   • In your application, click Certificates & secrets.
   • Verify your client secret hasn't expired.
   • If it expired, click + New client secret, enter a description and expiration period, copy the secret value, and update your connection with the new secret.
3. Verify the scope. Your connection must use the https://storage.azure.com/.default scope, which grants access to Azure Storage rather than to Microsoft Dynamics 365 directly.
4. Test the connection:
   • In Catalog Explorer, go to your connection.
   • Click Test connection to verify authentication.
   • If the test fails, review the error message for specific guidance.

If authentication still fails, use the following scripts in a Databricks notebook to isolate where it breaks.

Debugging scripts

Confirm that the client ID and the client secret are working correctly:

Python

%pip install azure-storage-blob==12.22.0 azure-identity==1.17.1 azure-storage-file-datalake==12.16.0
%restart_python

# Required libraries
from azure.identity import ClientSecretCredential
from azure.storage.blob import BlobServiceClient

# --- Your Azure Credentials and Storage Details ---
# Replace the placeholder values with your actual information

# Entra ID (Azure Active Directory) details
tenant_id = "<tenant-id>"
client_id = "<client-id>"
client_secret = "<client-secret>"

# Azure Storage details
storage_account_name = "<storage-account>"
container_name = "<container-name>"

# --- Script to List Folders ---

# Construct the Blob Storage URL
storage_account_url = f"https://{storage_account_name}.blob.core.windows.net"

# 1. Authenticate using the service principal
# The ClientSecretCredential object will handle the OAuth 2.0 flow
try:
    credential = ClientSecretCredential(tenant_id, client_id, client_secret)
except Exception as e:
    print(f"Error creating credential: {e}")
    # You may want to stop execution if credentials are not valid
    dbutils.notebook.exit("Failed to create credentials")

# 2. Create a BlobServiceClient
# This client is the main entry point for interacting with the Blob service
try:
    blob_service_client = BlobServiceClient(account_url=storage_account_url, credential=credential)
except Exception as e:
    print(f"Error creating BlobServiceClient: {e}")
    dbutils.notebook.exit("Failed to create BlobServiceClient")

# 3. Get a client for the specific container
try:
    container_client = blob_service_client.get_container_client(container_name)
except Exception as e:
    print(f"Error getting container client for '{container_name}': {e}")
    dbutils.notebook.exit("Failed to get container client")

# 4. List the "folders" in the container
# Folders in Blob Storage are virtual and are represented by prefixes in blob names.
# This code iterates through the blobs and extracts the top-level directory names.
try:
    blob_list = container_client.list_blobs()
    folder_list = set()

    for blob in blob_list:
        if "/" in blob.name:
            folder_name = blob.name.split('/')[0]
            folder_list.add(folder_name)

    # Print the list of unique folder names
    if folder_list:
        print(f"Folders found in container '{container_name}':")
        for folder in sorted(list(folder_list)):
            print(folder)
    else:
        print(f"No folders found in container '{container_name}'.")

except Exception as e:
    print(f"An error occurred while listing blobs: {e}")

Confirm that the Unity Catalog connection is able to dispatch the access token:

Python

import requests
import json
import os

# --- Databricks Notebook Context and API Token Retrieval ---
# This section securely retrieves the necessary API token from your Databricks environment
# to interact with Unity Catalog.
notebook_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
WORKSPACE_URL = notebook_context.apiUrl().get()
API_TOKEN = notebook_context.apiToken().get()

# --- Unity Catalog Connection Configuration ---
# IMPORTANT: Replace with the name of your Unity Catalog external connection to ADLS Gen2.
# This connection must be configured in Unity Catalog and granted necessary permissions
# to access your Azure Data Lake Storage Gen2 account.
CONNECTION_NAME = "<uc-connection-name>"

def get_uc_connection_access_token(connection_name: str, api_token: str) -> str:
    """
    Retrieves the access token for a Unity Catalog external connection to ADLS Gen2.
    """
    url = f"{WORKSPACE_URL}/api/2.1/unity-catalog/foreign-credentials"
    body = '{{"securables": [{{"type": "CONNECTION", "full_name": "{}"}}]}}'.format(
        connection_name
    )
    headers = {
        "Authorization": "Bearer {}".format(api_token),
        "Content-Type": "application/json",
    }
    response = requests.post(url=url, headers=headers, data=body)
    response.raise_for_status()  # Raise an exception for HTTP errors (e.g., 401, 403, 404)

    print(response.json())

    credentials = response.json()["securable_to_credentials"][0]["credentials"]["foreign_credential"]["options"]["options"]
    access_token = credentials["access_token"]
    return access_token

print(get_uc_connection_access_token(CONNECTION_NAME, API_TOKEN))

Verify that you can list container contents using the Unity Catalog connection:

Python

import requests
import json
import os
from datetime import datetime, timedelta
from azure.core.credentials import AccessToken, TokenCredential
from azure.storage.filedatalake import DataLakeServiceClient

notebook_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
WORKSPACE_URL = notebook_context.apiUrl().get()
API_TOKEN = notebook_context.apiToken().get()

CONNECTION_NAME = "<uc-connection-name>"
storage_account_name = "<storage-account-name>"
container_name = "<container-name>"

# --- Custom Credential Object for Azure SDK ---
class StaticTokenCredential(TokenCredential):
    """
    A simple credential class to wrap an existing access token for Azure SDKs.
    The expiration is set arbitrarily for the SDK's internal logic;
    your token's real expiry is governed by its issuer.
    """
    def __init__(self, token: str):
        self._token = AccessToken(token, expires_on=(datetime.now() + timedelta(hours=1)).timestamp())

    def get_token(self, *scopes, **kwargs) -> AccessToken:
        return self._token

# ==================== Main Logic to List Top-Level Folders ====================
try:
    # --- Input Validation ---
    if CONNECTION_NAME == "<uc-connection-name>":
        raise ValueError("Please update 'CONNECTION_NAME' with the name of your Unity Catalog connection.")
    if storage_account_name == "<storage-account-name>":
        raise ValueError("Please update 'storage_account_name' with your Azure Storage Account Name.")
    if container_name == "<container-name>":
        raise ValueError("Please update 'container_name' with your ADLS Gen2 Container Name.")

    print(f"Retrieving access token from Unity Catalog connection: '{CONNECTION_NAME}'...")
    access_token_string = get_uc_connection_access_token(CONNECTION_NAME, API_TOKEN)
    print("Access token retrieved successfully.")

    # 1. Initialize the DataLakeServiceClient using the retrieved token
    account_url = f"https://{storage_account_name}.dfs.core.windows.net"
    credential = StaticTokenCredential(access_token_string)
    datalake_service_client = DataLakeServiceClient(account_url=account_url, credential=credential)
    file_system_client = datalake_service_client.get_file_system_client(file_system=container_name)

    print(f"\nSuccessfully connected to ADLS Gen2 container: '{container_name}' in storage account: '{storage_account_name}'.")

    # 2. Get and print only the top-level directories
    print("\n--- Listing Top-Level Folders ---")

    all_paths = file_system_client.get_paths(path="/")

    for path in all_paths:
        print(path.name)
except Exception as e:
    print(f"An unexpected error occurred during execution.")
    print(f"Error details: {e}")

Cannot access ADLS Gen2 storage

Symptoms: Pipeline runs fail with "403 Forbidden" or "Access denied" errors, the connection test succeeds but the pipeline fails, or some tables work while others fail.

Cause: The Microsoft Entra ID application lacks the Storage Blob Data Contributor role, the role assignment is scoped to the wrong container or path, or network restrictions and storage account firewall rules block Databricks access.

Resolution:

1. Verify the role assignment:
   • In the Azure portal, go to your storage account.
   • Click Access Control (IAM), then Role assignments.
   • Verify your Microsoft Entra ID application has the Storage Blob Data Contributor role.
   • Verify the Scope is set to the entire storage account rather than a specific container.
2. Add the missing role:
   • Click + Add > Add role assignment.
   • Search for Storage Blob Data Contributor.
   • Click Next and add your application.
   • Click Review + assign, then wait 5 to 10 minutes for the permission changes to propagate.
3. Check network restrictions:
   • In your storage account, click Networking.
   • Verify that Public network access is set to Enabled from all networks or includes Databricks IP ranges.
   • If you use private endpoints, verify that Databricks can route to them.
4. Review the firewall rules:
   • In Networking, review the Firewall settings.
   • Add Databricks IP addresses to the allowlist if needed, or turn on Allow Azure services on the trusted services list.

Virtual entities not appearing in schema discovery

Symptoms: Virtual entities don't appear when listing tables, pipeline creation fails with "Table not found" errors for virtual entities, or only Dataverse-native tables are discoverable.

Cause: Virtual entities aren't configured or synchronized, Synapse Link isn't exporting them, or the virtual entity names don't match your table configuration.

Resolution:

1. Verify the virtual entity configuration:
   • In the Power Platform admin center, go to your environment.
   • Go to Settings > Product > Features.
   • Verify that Virtual entity data source is turned on.
   • Verify that your F&O virtual entities are configured and active.
2. Wait for synchronization. Virtual entities usually take up to 15 minutes to synchronize after configuration, but can take up to 30 minutes to appear in Dataverse. Check again after that period.
3. Verify that Synapse Link includes virtual entities:
   • In Power Apps, edit your Synapse Link connection.
   • Review the selected tables and verify that virtual entities are included in the export list.
   • Add missing virtual entities and save.
4. Check the virtual entity names. Virtual entity logical names might differ from F&O table names. In Power Apps, go to Tables, locate your virtual entities, then copy the exact Logical name and use it in your pipeline configuration.

Virtual entity schema changes not reflected

Symptoms: New columns in F&O don't appear in target Delta tables, pipeline runs succeed but data is incomplete, or schema drift warnings appear in pipeline logs.

Cause: The virtual entity metadata wasn't refreshed in Dataverse, Synapse Link is using a cached schema, or schema evolution limitations apply to virtual entities.

Resolution:

1. Refresh the virtual entity metadata:
   • In the Power Platform admin center, go to your environment.
   • Go to Virtual entities settings.
   • Click Refresh metadata for the affected virtual entities.
   • Wait up to 30 minutes for the metadata to synchronize.
2. Recreate the Synapse Link export:
   • In Power Apps, edit your Synapse Link connection.
   • Remove the affected virtual entity from the export list, save, and wait 5 minutes.
   • Add the virtual entity back to the export list, save, and wait for the initial export to complete.
3. Perform a full refresh. Virtual entity schema changes often require a full refresh. Stop your pipeline, delete the target Delta tables for the affected virtual entities, then restart the pipeline to recreate the tables with the updated schema.

The connector doesn't support automated schema evolution, so source schema changes require manual intervention. See Schema evolution.

Data type changes cause pipeline failures

Symptoms: Pipeline runs fail with "Type mismatch" or "Cannot cast" errors, ingestion stops after a Dynamics 365 update or configuration change, or error messages reference specific columns and data types.

Cause: A column data type changed in Dynamics 365 (for example, from string to integer), so the target Delta table schema is incompatible with the new data.

Resolution:

1. Identify the changed column:

   • Review the pipeline error logs to find the affected column and table.

   • In Power Apps, check the table definition for the column's current data type.

   • Compare it with your target Delta table schema:

     SQL

     DESCRIBE main.d365_data.tablename;

2. Perform a full refresh. Data type changes require a full refresh to recreate tables. Stop the affected pipeline, drop the target table, then restart the pipeline to recreate the table with the new schema:

   SQL

   DROP TABLE IF EXISTS main.d365_data.tablename;

3. Prevent future issues. Coordinate with your Dynamics 365 administrator before schema changes, test schema changes in a non-production environment first, and schedule full refreshes during maintenance windows.

important

The Dynamics 365 connector doesn't automatically handle data type changes. You must perform a full refresh to update table schemas. See Schema evolution.

Column renames not handled correctly

Symptoms: Renamed columns appear as new columns with NULL values, old column data is lost, or target tables have both the old and new column names.

Cause: The connector treats a column rename as a drop and add operation, with no automatic data migration from the old column name to the new one.

Resolution:

1. Before the rename occurs, coordinate with your Dynamics 365 administrator to perform a full refresh, which preserves historical data under the new column name.

2. After the rename occurs, perform a full refresh to reload all data with the new column names. Historical data then populates the new column.

3. If a full refresh isn't feasible, migrate the data manually:

   SQL

   -- Copy data from old column to new column
   UPDATE main.d365_data.tablename
   SET new_column_name = old_column_name
   WHERE new_column_name IS NULL AND old_column_name IS NOT NULL;
   
   -- Drop old column after verification
   ALTER TABLE main.d365_data.tablename DROP COLUMN old_column_name;

tip

To minimize disruption, plan column renames during scheduled maintenance windows and perform full refreshes immediately after.

Initial sync taking too long

Symptoms: A pipeline runs for hours without completing, the initial sync is slower than expected, or the pipeline times out or fails during the first run.

Cause: Large data volume in the source tables, a slow Synapse Link export, network bandwidth limitations, or too many tables in a single pipeline.

Resolution:

1. Start with fewer tables. Create a pipeline with 5 to 10 tables, verify it works correctly, then add more tables incrementally.
2. Wait for the Synapse Link export. Verify Synapse Link completed the initial export before you run the pipeline. In the Azure portal, verify that all table folders contain data files. The initial export can take hours for large datasets.
3. Split the work into multiple pipelines. Instead of one pipeline with 100 tables, create 5 pipelines with 20 tables each, then run them in parallel or sequentially based on resource availability. This reduces individual pipeline run time.
4. Monitor Azure bandwidth. Check Azure Storage metrics for throttling or bandwidth limits. If you're throttled, increase the storage account tier or add network capacity.

Incremental updates are slow

Symptoms: Incremental pipeline runs take longer than expected, pipeline performance degrades over time, or high change volume causes delays.

Cause: Large changelog files, too many folders accumulating in ADLS Gen2, or high-frequency changes creating many small folders.

Resolution:

1. Increase the pipeline run frequency. Smaller, more frequent changelog files process faster than large ones. For high-change environments, run every 5 to 15 minutes instead of hourly.
2. Review the Synapse Link export frequency. In Power Apps, check your Synapse Link export schedule. Synapse Link creates folders at regular intervals, typically every 5 to 15 minutes. Align your pipeline runs with that frequency.
3. Clean up old export folders. Configure lifecycle policies in your storage account to delete old exports, retaining only the past 7 to 30 days based on your recovery needs. This reduces the number of folders the connector must scan.
4. Reduce the change volume. Review Dynamics 365 processes that generate high-frequency updates, and batch updates where possible to reduce individual change events.

Missing records after ingestion

Symptoms: Row counts in target tables don't match the source tables, specific records are missing, or there are intermittent data gaps.

Cause: The Synapse Link export is incomplete, the pipeline skipped folders because of errors, filtering or permissions in the source system restrict visibility, or Synapse Link isn't exporting delete records.

Resolution:

1. Compare the record counts. Check the row count in Dynamics 365:

   SQL

   SELECT COUNT(*) FROM account;

   Then check the row count in the target Delta table and identify the magnitude of the discrepancy:

   SQL

   SELECT COUNT(*) FROM main.d365_data.account;

2. Verify the Synapse Link export is complete. In ADLS Gen2, verify that all table folders have recent timestamp folders. Look for gaps in the folder timestamps, which can indicate that Synapse Link stopped temporarily.

3. Check for filtering. Some Dynamics 365 tables have security filters that restrict which records are visible. Verify that your Synapse Link service account has permission to see all records, and check whether record ownership or business unit filters apply.

4. Perform a full refresh. If records are consistently missing, perform a full refresh to reload all data, then compare the counts again.

5. Check delete handling. If the missing records were deleted in Dynamics 365, verify that Synapse Link exports deletes. In Power Apps, check the Synapse Link settings for delete tracking. If deletes aren't exported, deleted records aren't reflected in your target tables.

Attachment metadata is incomplete

Symptoms: Attachment tables (for example, annotation or attachment) have missing or incomplete data, or the file names and metadata are incorrect.

Cause: Synapse Link isn't exporting the attachment tables, attachment permissions restrict visibility, or the attachment data is stored in different tables.

Resolution:

1. Verify that the attachment tables are exported. In Power Apps, check your Synapse Link connection and verify that the attachment-related tables are included, then add any that are missing and wait for the export:

   • annotation for notes and file attachments
   • attachment for email attachments
   • activitymimeattachment for activity attachments

2. Check the attachment permissions. Verify that your Synapse Link service account can read attachment records, because some attachments might be restricted by security roles.

3. Understand the metadata-only limitation. The connector ingests attachment metadata rather than file contents. To download files, use the Dynamics 365 Web API separately. See Attachments and files.

4. Verify that you're querying the correct attachment fields:

   SQL

   SELECT
     annotationid,
     objectid,
     subject,
     filename,
     filesize,
     mimetype,
     documentbody  -- Usually NULL; binary content not ingested
   FROM main.d365_data.annotation;

Additional support

If the guidance above doesn't resolve your issue, collect diagnostics before you contact support.

1. Collect diagnostics:
   • Pipeline ID and run timestamps.
   • Complete error messages from the pipeline logs.
   • Azure Synapse Link logs and status.
   • Screenshots of error messages or configurations.
2. Check for known issues. Review Known issues for known problems, and check the Databricks release notes for recent updates.
3. Create a support ticket. In your workspace, go to Help > Contact Support, then select Technical Support and provide a clear description of the issue, steps to reproduce it, the diagnostic information you collected, and the impact and urgency.
4. Provide feedback. Share feedback with your Databricks account team, including bugs, feature requests, or documentation issues.