Skip to main content

What is Lakeflow Designer?

Preview

This feature is in Public Preview.

Lakeflow Designer provides a visual canvas for analysts to perform data analytics, preparation, and basic automation. In Designer, you create visual data prep files, each made up of a sequence of operators (such as filter, join, and transform) arranged as a DAG to produce a result. All transformations are backed by code, which allows you to version files in Git and schedule them as jobs to seamlessly move to production.

With Lakeflow Designer, you can:

  • Design workflows using a drag-and-drop canvas.
  • Transform data without writing code using built-in operators for filtering, aggregating, joining, and reshaping data.
  • Use Genie Code to generate or refine transformations in your visual data prep using natural language.
  • Preview each interim step without running the entire pipeline.

LFD showing the features of a visual data prep.

In the image above, you can see:

  1. The drag-and-drop canvas
  2. Two operators
  3. A connection between two operators
  4. The output pane
  5. The data profile of the output
  6. The Genie Code prompt

Key concepts

The canvas

The canvas is the main workspace where you add, configure, and connect operators to build your visual data prep.

To navigate the canvas:

  • Pan: Hold Space while clicking and dragging, or slide two fingers on a trackpad.
  • Zoom: Pinch or stretch on a trackpad, or hold Ctrl and scroll.

The canvas toolbar is located in the header with canvas navigation tools: Zoom in icon. zoom in, Zoom out icon. zoom out, Fullscreen icon. fit view, DAG horizontal icon. auto-layout, and Drag icon. drag mode.

Right-click anywhere on the canvas to access common actions including adding operators, undo and redo, auto-layout, fit view, and opening the code pane.

You can also drag and drop an Excel or CSV file directly onto the canvas to create a Source operator for that file.

Operators

Operators are the building blocks of a visual data prep. An operator is an action such as a join, transform, or filter. You chain together operators on the canvas to build a workflow. Each operator is configurable based on its type.

LFD canvas showing operators on the DAG.

Lakeflow Designer includes built-in operators for common data transformation tasks. For details, see Built-in operators in Lakeflow Designer.

Connections

Connections define how data flows between operators. To create a connection, drag the small circle on the right edge of one operator to the small circle on the left edge of another operator. This specifies that data flows from the first operator into the second. Data flows from left to right through the visual data prep. Some operators, such as Join and Combine, accept multiple input connections.

LFD canvas showing a connection between two operators.

The output pane

The output pane appears at the bottom of the screen when you select an operator. Select any operator to see the results in the output window at the bottom of the screen. For most operator types, the input data is on the left and output data is on the right.

By default, operators process a limited sample of data. Use the Rows scanned dropdown in the output pane to control how many rows to process:

  • Rows scanned: Limit: Processes the first N input rows. Specify the number of rows in the field next to the dropdown.
  • Rows scanned: Max: Processes all input rows.

LFD output pane below the canvas.

warning

Running with Rows scanned: Max reruns all upstream operators with the complete, unbounded dataset and can take a long time.

In the output pane, you can choose to show details of the data in your output. In the top right corner of the output pane, choose the Sidebar icon. sidebar button to open the selection details. Select a subset of your data to see details about your selection.

Sidebar showing graphs and detail about the selected output data.

Genie Code

Genie Code lets you describe transformations in natural language. All interactions are agentic and use context from the Databricks platform.

LFD Genie Code prompt

Type a prompt to generate or modify transformations. To view the history of Genie Code interactions and see more detail on each response, open the Genie Code side pane by clicking Sparkle color icon. in the right sidebar. When the side pane is open, the in-canvas toolbar is minimized.

Next steps