Troubleshoot Microsoft Dynamics 365 ingestion
This feature is in Public Preview.
This page provides troubleshooting guidance for common issues with the Microsoft Dynamics 365 connector in Lakeflow Connect.
Azure Synapse Link issues
Synapse Link not exporting data
Symptoms:
- No folders appear in your ADLS Gen2 container after configuring Synapse Link.
- Folder timestamps stop updating.
- Pipeline runs fail with "No data found" errors.
Possible causes:
- Synapse Link connection is paused or stopped.
- Azure storage account permissions are incorrect.
- Selected tables aren't configured for export.
- Synapse Link encountered an error during export.
Solutions:
-
Check Synapse Link status:
- Sign in to Power Apps.
- Navigate to Azure Synapse Link in your environment.
- Verify your connection shows "Active" status.
- If paused, select Resume to restart export.
-
Verify storage permissions:
- In the Azure portal, navigate to your storage account.
- Select Access Control (IAM).
- Verify the Synapse Link managed identity has Storage Blob Data Contributor role.
- If missing, add the role assignment.
-
Check table configuration:
- In Power Apps, select your Synapse Link connection.
- Review the list of selected tables.
- Verify the tables you want to ingest are included.
- Add missing tables and wait for initial export (5-15 minutes).
-
Review Synapse Link logs:
- In Power Apps, select your Synapse Link connection.
- Select View logs or History.
- Look for error messages indicating export failures.
- Address specific errors (for example, storage quota, permissions).
FILE_PATH_DOES_NOT_EXIST error
Symptoms:
- Pipeline runs fail with
FILE_PATH_DOES_NOT_EXISTerror. - The connector can't find expected files in ADLS Gen2.
- Error indicates missing folders or file paths.
Possible causes:
- The Enable Incremental Update Folder Structure option is not enabled in Synapse Link configuration.
- The connector expects a specific folder structure that doesn't exist.
Solutions:
-
Enable incremental update folder structure:
- In Power Apps, edit your Synapse Link connection.
- Select Advanced to show advanced configuration settings.
- Toggle Enable Incremental Update Folder Structure to enable it.
- Save the configuration.
- Wait for Synapse Link to regenerate the folder structure (this may take several hours for large datasets).
-
Verify folder structure:
- In the Azure portal, navigate to your ADLS Gen2 storage account and container.
- Verify that table folders now contain timestamped subfolders (for example,
2025-12-19T10-30-00-000Z). - These timestamped folders contain the incremental updates the connector needs.
-
Retry the pipeline:
- After enabling the incremental folder structure, run your pipeline again.
- The connector should now find files in the expected locations.
Synapse Link is set up, but files are not appearing
Symptoms:
- Synapse Link shows "Active" status, but no CSV files appear in your ADLS Gen2 container.
- Pipeline runs fail with "No data found" or file format errors.
Possible causes:
- Synapse Link is configured to export data in Parquet format instead of CSV format.
- The Connect to your Azure Synapse Analytics Workspace option is selected, which forces Parquet export.
Solutions:
-
Verify file format in ADLS Gen2:
- In the Azure portal, navigate to your ADLS Gen2 storage account and container.
- Open a table folder and check the file extensions.
- If files have
.parquetextension instead of.csv, the format is incorrect.
-
Reconfigure Synapse Link for CSV export:
- The Dynamics 365 connector only supports CSV format. You must reconfigure Synapse Link.
- In Power Apps, edit or recreate your Synapse Link connection.
- Clear the Connect to your Azure Synapse Analytics Workspace checkbox.
- This ensures data is exported in CSV format instead of Parquet.
- Save the configuration and wait for Synapse Link to re-export data (this may take several hours for large datasets).
-
Verify CSV files are created:
- After reconfiguring, check your ADLS Gen2 container.
- Confirm new folders contain
.csvfiles. - Once CSV files appear, retry your pipeline run.
Synapse Link changelog missing VersionNumber
Symptoms:
- Pipeline runs fail with "VersionNumber field not found" errors.
- Incremental ingestion doesn't work.
- Only full refresh succeeds.
Possible causes:
- Synapse Link isn't configured to export changelogs.
- Change tracking isn't enabled for tables.
- Synapse Link version is outdated.
Solutions:
-
Enable change tracking:
- In Power Apps, edit your Synapse Link connection.
- Ensure Enable change tracking is selected.
- Save and wait for Synapse Link to regenerate exports (up to 30 minutes).
-
Verify changelog files:
- In the Azure portal, navigate to your ADLS Gen2 container.
- Open a table folder and locate the
SynapseLinksubfolder. - Open a recent changelog file (CSV or JSON).
- Verify the file contains a
VersionNumbercolumn. - If missing, contact Microsoft support to enable change tracking.
-
Update Synapse Link:
- Verify you're using Azure Synapse Link for Dataverse version 1.0 or later.
- Older versions might not support
VersionNumber. - Update Synapse Link to the latest version if needed.
-
Perform a full refresh:
- If change tracking can't be enabled, you can only use full refresh mode.
- Note that full refresh reloads all data on every run, which is slower and more expensive.
Authentication issues
Entra ID authentication fails
Symptoms:
- Pipeline creation fails with "Authentication failed" errors.
- Connection test fails in Catalog Explorer.
- Pipeline runs fail with "401 Unauthorized" errors.
Possible causes:
- Incorrect tenant ID, client ID, or client secret.
- Client secret expired.
- Application doesn't have required permissions.
- Incorrect scope specified.
Solutions:
-
Verify authentication parameters:
- In the Azure portal, navigate to Microsoft Entra ID > App registrations.
- Locate your application and verify:
- Application (client) ID matches your connection configuration.
- Directory (tenant) ID matches your connection configuration.
- Copy the correct values and update your connection if needed.
-
Check client secret expiration:
- In your application, select Certificates & secrets.
- Verify your client secret hasn't expired.
- If expired, create a new secret:
- Select + New client secret.
- Enter a description and expiration period.
- Copy the secret value.
- Update your connection with the new secret.
-
Verify scope:
- Ensure your connection uses the correct scope:
https://storage.azure.com/.default - This scope grants access to Azure Storage, not Dynamics 365 directly.
- Ensure your connection uses the correct scope:
-
Test connection:
- In Catalog Explorer, navigate to your connection.
- Select Test connection to verify authentication.
- If the test fails, review the error message for specific guidance.
Cannot access ADLS Gen2 storage
Symptoms:
- Pipeline runs fail with "403 Forbidden" or "Access denied" errors.
- Connection test succeeds but pipeline fails.
- Some tables work but others fail.
Possible causes:
- Entra ID application doesn't have Storage Blob Data Contributor role.
- Role assignment is scoped to wrong container or path.
- Network restrictions block access.
- Storage account firewall rules block Databricks access.
Solutions:
-
Verify role assignment:
- In the Azure portal, navigate to your storage account.
- Select Access Control (IAM).
- Select Role assignments.
- Verify your Entra ID application has Storage Blob Data Contributor role.
- Check the Scope is set to the entire storage account, not a specific container.
-
Add missing role:
- Select + Add > Add role assignment.
- Search for Storage Blob Data Contributor.
- Select Next and add your application.
- Select Review + assign.
- Wait 5-10 minutes for permission changes to propagate.
-
Check network restrictions:
- In your storage account, select Networking.
- Verify Public network access is Enabled from all networks or includes Databricks IP ranges.
- If using private endpoints, ensure Databricks can route to them.
-
Review firewall rules:
- In Networking, review Firewall settings.
- Add Databricks IP addresses to the allowlist if needed.
- Alternatively, enable Allow Azure services on the trusted services list.
Virtual entity issues
Virtual entities not appearing in schema discovery
Symptoms:
- Virtual entities don't appear when listing tables.
- Pipeline creation fails with "Table not found" errors for virtual entities.
- Only Dataverse-native tables are discoverable.
Possible causes:
- Virtual entities aren't configured or synchronized.
- Synapse Link isn't exporting virtual entities.
- Virtual entity names don't match table configuration.
Solutions:
-
Verify virtual entity configuration:
- In Power Platform admin center, navigate to your environment.
- Go to Settings > Product > Features.
- Verify Virtual entity data source is enabled.
- Check that your F&O virtual entities are configured and active.
-
Wait for synchronization:
- Virtual entities can take up to 15 minutes to synchronize after configuration.
- Allow up to 30 minutes for virtual entities to appear in Dataverse.
- Check again after the synchronization period.
-
Verify Synapse Link includes virtual entities:
- In Power Apps, edit your Synapse Link connection.
- Review the selected tables.
- Ensure virtual entities are included in the export list.
- Add missing virtual entities and save.
-
Check virtual entity names:
- Virtual entity logical names might differ from F&O table names.
- In Power Apps, go to Tables and locate your virtual entities.
- Copy the exact Logical name and use it in your pipeline configuration.
Virtual entity schema changes not reflected
Symptoms:
- New columns in F&O don't appear in target Delta tables.
- Pipeline runs succeed but data is incomplete.
- Schema drift warnings in pipeline logs.
Possible causes:
- Virtual entity metadata not refreshed in Dataverse.
- Synapse Link using cached schema.
- Schema evolution limitations for virtual entities.
Solutions:
-
Refresh virtual entity metadata:
- In Power Platform admin center, navigate to your environment.
- Go to Virtual entities settings.
- Select Refresh metadata for affected virtual entities.
- Wait up to 30 minutes for metadata to synchronize.
-
Recreate Synapse Link export:
- In Power Apps, edit your Synapse Link connection.
- Remove the affected virtual entity from the export list.
- Save and wait 5 minutes.
- Add the virtual entity back to the export list.
- Save and wait for initial export to complete.
-
Perform a full refresh:
- Virtual entity schema changes often require a full refresh.
- Stop your pipeline.
- Delete target Delta tables for affected virtual entities.
- Restart the pipeline to recreate tables with updated schema.
Schema evolution issues
Data type changes cause pipeline failures
Symptoms:
- Pipeline runs fail with "Type mismatch" or "Cannot cast" errors.
- Ingestion stops after a D365 update or configuration change.
- Error messages reference specific columns and data types.
Possible causes:
- Column data type changed in D365 (for example, string to integer).
- Target Delta table has old schema incompatible with new data.
- Synapse Link exports data in new format but target table expects old format.
Solutions:
-
Identify the changed column:
- Review pipeline error logs to find the affected column and table.
- In Power Apps, check the table definition for the column's current data type.
- Compare with your target Delta table schema:
SQL
DESCRIBE main.d365_data.tablename;
-
Perform a full refresh:
- Data type changes require a full refresh to recreate tables.
- Stop the affected pipeline.
- Drop the target table:
SQL
DROP TABLE IF EXISTS main.d365_data.tablename; - Restart the pipeline to recreate the table with the new schema.
-
Prevent future issues:
- Coordinate with your D365 administrator before schema changes.
- Test schema changes in a non-production environment first.
- Schedule full refreshes during maintenance windows.
The Dynamics 365 connector doesn't automatically handle data type changes. You must perform a full refresh to update table schemas. See limitations for details.
Column renames not handled correctly
Symptoms:
- Renamed columns appear as new columns with NULL values.
- Old column data is lost.
- Target tables have both old and new column names.
Possible causes:
- The connector treats column renames as a drop and add operation.
- No automatic data migration from old to new column name.
Solutions:
-
Before rename occurs:
- If possible, coordinate with your D365 administrator to perform a full refresh before the rename.
- This preserves historical data in the new column name.
-
After rename occurs:
- Perform a full refresh to reload all data with new column names.
- Historical data will populate the new column.
-
Manual migration (if full refresh isn't feasible):
-
If you can't perform a full refresh, manually migrate data:
SQL-- Copy data from old column to new column
UPDATE main.d365_data.tablename
SET new_column_name = old_column_name
WHERE new_column_name IS NULL AND old_column_name IS NOT NULL;
-- Drop old column after verification
ALTER TABLE main.d365_data.tablename DROP COLUMN old_column_name;
-
To minimize disruption, plan column renames during scheduled maintenance windows and perform full refreshes immediately after.
Performance issues
Initial sync taking too long
Symptoms:
- Pipeline runs for hours without completing.
- Initial sync slower than expected.
- Pipeline times out or fails during first run.
Possible causes:
- Large data volume in source tables.
- Slow Synapse Link export.
- Network bandwidth limitations.
- Too many tables in a single pipeline.
Solutions:
-
Start with fewer tables:
- Create a pipeline with a small subset of tables (5-10 tables).
- Verify the pipeline works correctly.
- Add more tables incrementally after initial validation.
-
Wait for Synapse Link export:
- Verify Synapse Link completed the initial export before running the pipeline.
- In the Azure portal, check that all table folders contain data files.
- Initial Synapse Link export can take hours for large datasets.
-
Split into multiple pipelines:
- Instead of one pipeline with 100 tables, create 5 pipelines with 20 tables each.
- Run pipelines in parallel or sequentially based on resource availability.
- This approach reduces individual pipeline run time.
-
Monitor Azure bandwidth:
- Check Azure Storage metrics for throttling or bandwidth limits.
- If throttled, increase storage account tier or add more network capacity.
Incremental updates slow
Symptoms:
- Incremental pipeline runs take longer than expected.
- Pipeline performance degrades over time.
- High change volume causes delays.
Possible causes:
- Large changelog files.
- Too many folders accumulating in ADLS Gen2.
- High-frequency changes creating many small folders.
Solutions:
-
Increase pipeline run frequency:
- If changelogs are large, run pipelines more frequently.
- Smaller, more frequent changelog files process faster than large ones.
- For high-change environments, run every 5-15 minutes instead of hourly.
-
Review Synapse Link export frequency:
- In Power Apps, check your Synapse Link export schedule.
- Synapse Link creates folders at regular intervals (typically every 5-15 minutes).
- Align pipeline runs with Synapse Link export frequency.
-
Clean up old export folders:
- Configure lifecycle policies in your storage account to delete old exports.
- Retain only the past 7-30 days of exports based on your recovery needs.
- This reduces the number of folders the connector must scan.
-
Reduce change volume:
- Review D365 processes that generate high-frequency updates.
- Batch updates where possible to reduce individual change events.
- Consider if all updates need to be captured (some might be transient).
Data quality issues
Missing records after ingestion
Symptoms:
- Row counts in target tables don't match source tables.
- Specific records missing from target tables.
- Intermittent data gaps.
Possible causes:
- Synapse Link export incomplete.
- Pipeline skipped folders due to errors.
- Filtering or permissions in source system.
- Delete records not exported by Synapse Link.
Solutions:
-
Compare record counts:
- Check row counts in D365:
SQL
-- In D365/Dataverse
SELECT COUNT(*) FROM account; - Check row counts in target Delta table:
SQL
-- In Databricks
SELECT COUNT(*) FROM main.d365_data.account; - Identify the magnitude of the discrepancy.
- Check row counts in D365:
-
Verify Synapse Link completeness:
- In ADLS Gen2, check that all table folders have recent timestamp folders.
- Look for gaps in folder timestamps that might indicate missed exports.
- If gaps exist, Synapse Link might have stopped temporarily.
-
Check for filtering:
- Some D365 tables have security filters that restrict visible records.
- Verify your Synapse Link service account has permissions to see all records.
- Check if record ownership or business unit filters apply.
-
Perform a full refresh:
- If records are consistently missing, perform a full refresh.
- This reloads all data and ensures completeness.
- Compare counts again after the full refresh.
-
Check delete handling:
- If missing records were deleted in D365, verify your Synapse Link exports deletes.
- In Power Apps, check Synapse Link settings for delete tracking.
- If deletes aren't exported, deleted records won't be reflected in target tables.
Attachment metadata incomplete
Symptoms:
- Attachment tables (for example,
annotation,attachment) have missing or incomplete data. - File names or metadata incorrect.
Possible causes:
- Synapse Link doesn't export attachment tables.
- Attachment permissions restrict visibility.
- Attachment data stored in different tables.
Solutions:
-
Verify attachment tables are exported:
- In Power Apps, check your Synapse Link connection.
- Ensure attachment-related tables are included:
annotation(for notes and file attachments)attachment(for email attachments)activitymimeattachment(for activity attachments)
- Add missing tables and wait for export.
-
Check attachment permissions:
- Verify your Synapse Link service account can read attachment records.
- Some attachments might be restricted by security roles.
- Adjust permissions if needed.
-
Understand metadata-only limitation:
- The Dynamics 365 connector ingests attachment metadata, not file contents.
- To download files, use the Dynamics 365 Web API separately.
- See limitations for details.
-
Query attachment metadata:
- Verify you're querying the correct attachment fields:
SQL
SELECT
annotationid,
objectid,
subject,
filename,
filesize,
mimetype,
documentbody -- Usually NULL; binary content not ingested
FROM main.d365_data.annotation;
- Verify you're querying the correct attachment fields:
Additional support
If you can't resolve issues using this guide:
-
Collect diagnostics:
- Pipeline ID and run timestamps.
- Complete error messages from pipeline logs.
- Azure Synapse Link logs and status.
- Screenshots of error messages or configurations.
-
Check for known issues:
- Review limitations for known problems.
- Check Databricks release notes for recent updates.
-
Create a support ticket:
- In your workspace, go to Help > Contact Support.
- Select Technical Support and provide:
- Clear description of the issue.
- Steps to reproduce the problem.
- All diagnostic information collected above.
- Impact and urgency of the issue.
-
Provide feedback:
- For Beta, share feedback with your Databricks account team.
- Report bugs, feature requests, or documentation issues.
Synapse Link is set up, but I’m not seeing files
First, confirm that the files are written in CSV format by Synapse Link (not Parquet). To do this, clear the Connect to your Azure Synapse Analytics Workspace checkbox when linking to Azure Synapse Link.
Credentials/login error
Try the following debugging scripts:
Confirm that the client ID and the client secret are working correctly:
%pip install azure-storage-blob azure-identity azure-storage-file-datalake
%restart_python
# Required libraries
from azure.identity import ClientSecretCredential
from azure.storage.blob import BlobServiceClient
# --- Your Azure Credentials and Storage Details ---
# Replace the placeholder values with your actual information
# Entra ID (Azure Active Directory) details
tenant_id = "<tenant-id>"
client_id = "<client-id>"
client_secret = "<client-secret>"
# Azure Storage details
storage_account_name = "<storage-account>"
container_name = "<container-name>"
# --- Script to List Folders ---
# Construct the Blob Storage URL
storage_account_url = f"https://{storage_account_name}.blob.core.windows.net"
# 1. Authenticate using the service principal
# The ClientSecretCredential object will handle the OAuth 2.0 flow
try:
credential = ClientSecretCredential(tenant_id, client_id, client_secret)
except Exception as e:
print(f"Error creating credential: {e}")
# You may want to stop execution if credentials are not valid
dbutils.notebook.exit("Failed to create credentials")
# 2. Create a BlobServiceClient
# This client is the main entry point for interacting with the Blob service
try:
blob_service_client = BlobServiceClient(account_url=storage_account_url, credential=credential)
except Exception as e:
print(f"Error creating BlobServiceClient: {e}")
dbutils.notebook.exit("Failed to create BlobServiceClient")
# 3. Get a client for the specific container
try:
container_client = blob_service_client.get_container_client(container_name)
except Exception as e:
print(f"Error getting container client for '{container_name}': {e}")
dbutils.notebook.exit("Failed to get container client")
# 4. List the "folders" in the container
# Folders in Blob Storage are virtual and are represented by prefixes in blob names.
# This code iterates through the blobs and extracts the top-level directory names.
try:
blob_list = container_client.list_blobs()
folder_list = set()
for blob in blob_list:
if "/" in blob.name:
folder_name = blob.name.split('/')[0]
folder_list.add(folder_name)
# Print the list of unique folder names
if folder_list:
print(f"Folders found in container '{container_name}':")
for folder in sorted(list(folder_list)):
print(folder)
else:
print(f"No folders found in container '{container_name}'.")
except Exception as e:
print(f"An error occurred while listing blobs: {e}")
Confirm that the Unity Catalog connection is able to dispatch the access token:
import requests
import json
import os
# --- Databricks Notebook Context and API Token Retrieval ---
# This section securely retrieves the necessary API token from your Databricks environment
# to interact with Unity Catalog.
notebook_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
WORKSPACE_URL = notebook_context.apiUrl().get()
API_TOKEN = notebook_context.apiToken().get()
# --- Unity Catalog Connection Configuration ---
# IMPORTANT: Replace with the name of your Unity Catalog external connection to ADLS Gen2.
# This connection must be configured in Unity Catalog and granted necessary permissions
# to access your Azure Data Lake Storage Gen2 account.
# Example: CONNECTION_NAME = "my_adls_gen2_connection"
CONNECTION_NAME = "<uc-connection-name>"
def get_uc_connection_access_token(connection_name: str, api_token: str) -> str:
"""
Retrieves the access token for a Unity Catalog external connection to ADLS Gen2.
"""
url = f"{WORKSPACE_URL}/api/2.1/unity-catalog/foreign-credentials"
body = '{{"securables": [{{"type": "CONNECTION", "full_name": "{}"}}]}}'.format(
connection_name
)
headers = {
"Authorization": "Bearer {}".format(api_token),
"Content-Type": "application/json",
}
response = requests.post(url=url, headers=headers, data=body)
response.raise_for_status() # Raise an exception for HTTP errors (e.g., 401, 403, 404)
print(response.json())
credentials = response.json()["securable_to_credentials"][0]["credentials"]["foreign_credential"]["options"]["options"]
access_token = credentials["access_token"]
return access_token
print(get_uc_connection_access_token(CONNECTION_NAME, API_TOKEN))
Verify we are able to list container contents using UC connection
import requests
import json
import os
from datetime import datetime, timedelta
from azure.core.credentials import AccessToken, TokenCredential
from azure.storage.filedatalake import DataLakeServiceClient
notebook_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
WORKSPACE_URL = notebook_context.apiUrl().get()
API_TOKEN = notebook_context.apiToken().get()
CONNECTION_NAME = "<uc-connection-name>"
storage_account_name = "<storage-account-name>"
container_name = "<container-name>"
# --- Custom Credential Object for Azure SDK ---
class StaticTokenCredential(TokenCredential):
"""
A simple credential class to wrap an existing access token for Azure SDKs.
The expiration is set arbitrarily for the SDK's internal logic;
your token's real expiry is governed by its issuer.
"""
def __init__(self, token: str):
self._token = AccessToken(token, expires_on=(datetime.now() + timedelta(hours=1)).timestamp())
def get_token(self, *scopes, **kwargs) -> AccessToken:
return self._token
# ==================== Main Logic to List Top-Level Folders ====================
try:
# --- Input Validation ---
if CONNECTION_NAME == "<YOUR_UNITY_CATALOG_CONNECTION_NAME>":
raise ValueError("Please update 'CONNECTION_NAME' with the name of your Unity Catalog connection.")
if storage_account_name == "<YOUR_STORAGE_ACCOUNT_NAME>":
raise ValueError("Please update 'storage_account_name' with your Azure Storage Account Name.")
if container_name == "<YOUR_CONTAINER_NAME>":
raise ValueError("Please update 'container_name' with your ADLS Gen2 Container Name.")
print(f"Retrieving access token from Unity Catalog connection: '{CONNECTION_NAME}'...")
access_token_string = get_uc_connection_access_token(CONNECTION_NAME, API_TOKEN)
print("Access token retrieved successfully.")
# 1. Initialize the DataLakeServiceClient using the retrieved token
account_url = f"https://{storage_account_name}.dfs.core.windows.net"
credential = StaticTokenCredential(access_token_string)
datalake_service_client = DataLakeServiceClient(account_url=account_url, credential=credential)
file_system_client = datalake_service_client.get_file_system_client(file_system=container_name)
print(f"\nSuccessfully connected to ADLS Gen2 container: '{container_name}' in storage account: '{storage_account_name}'.")
# 2. Get and print only the top-level directories
print("\n--- Listing Top-Level Folders ---")
all_paths = file_system_client.get_paths(path="/")
for path in all_paths:
print(path.name)
except Exception as e:
print(f"An unexpected error occurred during execution.")
print(f"Error details: {e}")
Error: The selected storage account has restricted network access...
The selected storage account has restricted network access. To proceed, please setup an enterprise policy and connect it to your Dataverse environment. Once done, please enable the 'Select Enterprise Policy with Managed Service Identity' option below.
This can happen when your ADLS staging location is secured by a firewall, and Dataverse can’t reach it. Follow Use managed identities for Azure with your Azure data lake storage in the Microsoft documentation to set up a managed identity (formerly managed service identity) to access your data.
Error: FILE_PATH_DOES_NOT_EXIST
This typically happens when Enable Incremental Update Folder Structure wasn't toggled when Synapse Link was configured. As a result, the connector doesn't find the file where it expects to.