Skip to main content
Last updated on

Ingest data into Lakeflow Designer

Preview

This feature is in Public Preview.

This page describes the available options for bringing data into a Visual data prep in Lakeflow Designer. Designer can work with any data accessible through Databricks.

All data ingestion in Designer starts with the Source operator. When you open a Source operator's configuration pane, you have the following options.

Browse existing tables​

Click Browse existing to open the asset selector, where you can:

  • Browse tables and volumes in Unity Catalog.
  • Browse files in the workspace file system.

Use this option when your data is already in Databricks.

Drag and drop a local file​

Drag a CSV or Excel file from your local machine directly onto the canvas. Designer uploads the file to the workspace file system and automatically creates a Source operator.

note

Drag and drop of Excel files requires Excel File Format Support to be enabled. See Read Excel files.

Create a table from a file​

Select Create table from file in the Source configuration pane to upload a local CSV or Excel file and create a Unity Catalog table from it. Use this option when you want the data persisted as a managed table rather than stored as a workspace file. Additionally, for large amounts of data, this provides the best performance.

Import from Google Drive​

To import a file from Google Drive:

  1. In the Source configuration pane, select Import from Google Drive.
  2. Select a Unity Catalog connection that has access to Google Drive.
  3. Provide the file URL.

This requires a Unity Catalog connection configured for Google Drive access. See Ingest files from Google Drive for how to set up connections.

Import from SharePoint​

To import a file from SharePoint:

  1. In the Source configuration pane, select Import from SharePoint.
  2. Select a Unity Catalog connection that has access to your SharePoint site.
  3. Provide the direct file link.
warning

To get the correct URL, right-click the file in SharePoint, select Preview, and copy the URL from the browser address bar. Do not use a sharing link.

This requires a Unity Catalog connection configured for SharePoint access. See Ingest files from SharePoint for how to set up connections.

Use Lakeflow Connect for additional sources​

If a data source is not available directly in Designer, check Lakeflow Connect for a managed connector. Lakeflow Connect supports ingestion from SaaS applications (Salesforce, Workday, HubSpot, and more), databases, cloud storage, and streaming sources. After you ingest data with Lakeflow Connect into a Unity Catalog table, you can use it as a Source in Designer.

Next steps​

On this page