Skip to main content

What is Lakeflow Designer?

Preview

This feature is in Public Preview.

Lakeflow Designer provides a visual canvas for analysts to perform data analytics, preparation, and basic automation. In Designer, you create Visual data preps, each made up of a sequence of operators (such as filter, join, and transform) arranged as a DAG to produce a result. All transformations are backed by code, which supports moving workflows to production.

With Lakeflow Designer, you can:

  • Design Visual data preps using the drag-and-drop canvas.
  • Transform data without writing code using built-in operators for filtering, aggregating, joining, and reshaping data.
  • Use Genie Code to generate or refine transformations in your Visual data prep using natural language.
  • Preview each interim step without running the entire pipeline.

LFD showing the features of a Visual data prep.

In the image above, you can see:

  1. The drag-and-drop canvas
  2. Two operators
  3. A connection between two operators
  4. The output pane
  5. The data profile of the output
  6. The Genie Code prompt

Key concepts

The canvas

The canvas is the main workspace where you add, configure, and connect operators to build your Visual data prep.

To navigate the canvas:

  • Pan: Hold Space while clicking and dragging, or slide two fingers on a trackpad.
  • Zoom: Pinch or stretch on a trackpad, or hold Ctrl and scroll.

The canvas has a toolbar in the bottom-left with canvas navigation tools: Zoom in icon. zoom in, Zoom out icon. zoom out, Fullscreen icon. fit view, DAG horizontal icon. auto-layout, and Drag icon. drag mode.

Right-click anywhere on the canvas to access common actions including adding operators, undo and redo, auto-layout, fit view, and opening the code pane.

You can also drag and drop an Excel or CSV file directly onto the canvas to create a Source operator for that file.

Operators

Operators are the building blocks of a Visual data prep. An operator is an action such as a join, transform, or filter. You chain together operators on the canvas to build a workflow. Each operator is configurable based on its type.

LFD canvas showing operators on the DAG.

Lakeflow Designer includes built-in operators for common data transformation tasks. For details, see Built-in operators in Lakeflow Designer.

Connections

Connections define how data flows between operators. To create a connection, drag the small circle on the right edge of one operator to the small circle on the left edge of another operator. This specifies that data flows from the first operator into the second. Data flows from left to right through the Visual data prep. Some operators, such as Join and Combine, accept multiple input connections.

LFD canvas showing a connection between two operators.

The output pane

The output pane appears at the bottom of the screen when you select an operator. Select any operator to see the results in the output window at the bottom of the screen. For most operator types, the input data is on the left and output data is on the right.

By default, operators run on a sample of the data of up to 1,000 rows. To run with the full dataset, click Sample dataset in the output pane and switch to Full dataset.

LFD output pane below the canvas.

warning

Running with the full dataset reruns all upstream operators with the complete, unbounded dataset and can take a long time.

In the output pane, you can choose to show details of the data in your output. In the top right corner of the output pane, choose the Sidebar icon. sidebar button to open the selection details. Select a subset of your data to see details about your selection.

Sidebar showing graphs and detail about the selected output data.

Genie Code

Genie Code lets you describe transformations in natural language. All interactions are agentic and use context from the Databricks platform.

LFD Genie Code prompt

Type a prompt to generate or modify transformations. To view the history of Genie Code interactions and see more detail on each response, open the Genie Code side pane by clicking Sparkle color icon. in the right sidebar. When the side pane is open, the in-canvas toolbar is minimized.

Next steps