What is Lakeflow Designer?

Lakeflow Designer provides a visual canvas for analysts to perform data analytics, preparation, and basic automation. In Designer, you create visual data prep files, each made up of a sequence of operators (such as filter, join, and transform) arranged as a graph to produce a result. All transformations are backed by code, which allows you to version files in Git and schedule them as jobs to seamlessly move to production.

With Lakeflow Designer, you can:

• Design workflows using a drag-and-drop canvas.
• Transform data without writing code using built-in operators for filtering, aggregating, joining, and reshaping data.
• Use Genie Code to generate or refine transformations in your visual data prep using natural language.
• Preview each interim step without running the entire pipeline.

In the image above, you can see:

1. The left panel, with the Operators, Table of contents, and Parameters menus
2. An operator
3. A connection between two operators
4. The drag-and-drop canvas
5. The Genie Code prompt
6. The output pane
7. The data profile of the output

Key concepts

The left panel

The left panel brings together the menus you use to build and navigate a visual data prep:

• Operators: Browse and search the built-in operators. Drag and drop operator types from the list onto the canvas to add them.
• Parameters: Define and manage parameters for the visual data prep.
• Table of contents: Find and navigate to any operator in the visual data prep, and view the code generated for each one. See The table of contents.

The canvas

The canvas is the main workspace where you add, configure, and connect operators to build your visual data prep.

To navigate the canvas:

• Pan: Hold Space while clicking and dragging, or slide two fingers on a trackpad.
• Zoom: Pinch or stretch on a trackpad, or hold Ctrl and scroll.

The canvas toolbar is located in the header with canvas navigation tools: zoom in, zoom out, fit view, auto-layout, and drag mode.

Right-click anywhere on the canvas to access common actions including adding operators, undo and redo, auto-layout, fit view, and opening the code pane.

To quickly access editor actions, open the command palette by pressing Cmd+Shift+P on macOS or Ctrl+Shift+P on Windows.

You can also drag and drop an Excel or CSV file directly onto the canvas to create a Source operator for that file.

Operators

Operators are the building blocks of a visual data prep. An operator is an action such as a join, transform, or filter. You chain together operators on the canvas to build a workflow. Each operator is configurable based on its type. Operators display an AI-generated description of their effect; editing the description reconfigures the operator.

Select an operator to highlight its lineage on the canvas. The connections tracing back to its upstream sources and forward to its downstream sinks are highlighted, so you can follow how data reaches and flows from that operator.

Lakeflow Designer includes built-in operators for common data transformation tasks. For details, see Built-in operators in Lakeflow Designer.

Connections

Connections define how data flows between operators. To create a connection, drag the port on the right edge of one operator to the port on the left edge of another operator. This specifies that data flows from the first operator into the second. Data flows from left to right through the visual data prep. Some operators, such as Join and Combine, accept multiple input connections.

Node ports provide large connection targets and clear visual cues, so operators are easy to connect.

An operator's ports also signal its configuration status:

• An unconfigured operator has a dashed border.
• A required input port that isn't connected is highlighted in yellow.

The output pane

The output pane appears at the bottom of the screen when you select an operator. Select any operator to see the results in the output window at the bottom of the screen. For most operator types, the input data is on the left and output data is on the right. Operators that produce non-table results, such as plots, HTML, or images, render those outputs directly in the output pane.

Use the view control in the output pane to switch between input and output (the default), input only, or output only. In the combined view, drag the divider to resize the input and output panes.

You can also filter and transform data directly in the results table, then apply those changes to the canvas as operators. See Transform data directly in the results table.

Designer previews operators automatically. Use the Preview menu in the toolbar to set the default number of input rows to process for previews or the Rows scanned menu in the output pane to override for a specific operator. See Preview results.

The data profile

To see details about part of your output, select the data you want to inspect, then click sidebar button in the top-right corner of the output pane to open the details panel.

Genie Code

Genie Code enables you to describe transformations in natural language. All interactions are agentic and use context from the Databricks platform. See Genie Code.

Type a prompt to generate or modify transformations. To view the history of Genie Code interactions and see more detail on each response, open the Genie Code side pane by clicking in the right sidebar. When the side pane is open, the in-canvas toolbar is minimized. Genie Code shows a one-line summary of its most recent edit above the input box.

The table of contents

The Table of contents menu in the left panel lists every operator in the visual data prep. Search by operator name or by a value in an operator's configuration, such as a column name. For configuration matches, Designer shows the matching configuration paths below the operator. Select an operator to focus it on the canvas. Every visual data prep is backed by generated code. To inspect the code for an operator, select it in the table of contents, then expand the Generated code section to view its underlying code.

Parameters

Parameters are named values defined for the visual data prep as a whole. You can reference them in SQL and Python operator code or use them in supported configuration fields for Source and Output operators. To manage parameters, select Parameters in the left panel.

Each parameter has a value that you set when you define it. When you schedule the visual data prep to run, you can optionally override these values for each schedule. For example, you might schedule the same visual data prep to run every day at noon with an environment parameter set to test, and again at 2 PM with environment set to production.

Reference a parameter from an operator as follows:

• SQL operator: Use named parameter marker syntax, such as :environment. See Use named parameter markers.
• Python operator: Call dbutils.widgets.get(), such as dbutils.widgets.get("environment"). See Widgets tool (dbutils.widgets).
• Source or Output operator: Open the operator's configuration pane and select the Parameters button in the upper-right corner. Select a configuration field, then enter an expression that references a parameter, such as :source_table. Source operators support table, file path, and file format fields. Output operators support catalog, schema, table, view, volume, file name, and write mode fields.

In a parameter expression for a Source or Output operator, use :parameter_name to replace an entire value. To combine parameters with fixed text, enclose the text in single quotes and use ||. For example, the following Source Table expression constructs a fully qualified table name:

:catalog || '.' || :schema || '.' || :table

When you open a SQL or Python operator for editing, Lakeflow Designer shows an example of how to reference the available parameters above the source editor.

Additional resources

• Create a visual data prep in Lakeflow Designer
• Ingest data into Lakeflow Designer
• Built-in operators in Lakeflow Designer