Custom operators in Lakeflow Designer

Lakeflow Designer lets you create custom operators that appear directly in the canvas alongside built-in operators. Use them to extend Lakeflow Designer with your own business logic, calculations, or integrations. Custom operators were previously called user-defined operators (UDOs).

Custom operators come in two scopes:

• Local: The operator belongs to one visual data prep file. Describe the transformation, and Genie Code generates the operator. See Create a local custom operator with Genie Code.
• Shared: The operator is registered in the workspace, so you and other users can add it to any visual data prep file. Its logic is a Python function or a Unity Catalog function, described in YAML. Write it yourself, or ask Genie Code to create it. See How custom operators work.

Create a local custom operator with Genie Code

When Genie Code is available in your workspace, you can create a local custom operator from the Lakeflow Designer canvas:

1. In the Operators pane, select Create custom operator.

2. Enter a description and, optionally, a name. Select Generate preview.

   prompt

   For example, enter the following description:

   Count the business days between two date columns, excluding weekends. Let me choose the start and end date columns and name the output column.

3. Review the generated configuration and Python function, then select Approve and create.

Designer stores the local operator's definition in the current file and adds it to the Operators pane. Use the operator's actions menu to edit or delete the definition. You must remove every instance of the operator from the canvas before you can delete its definition.

Choose a shared operator type

Shared custom operators have three types:

• python-run-function: A standalone YAML file with inline Python stored in the workspace. Best for DataFrame-level transformations and external integrations. Permissions are managed at the workspace file level.
• uc-udf: Wraps a Unity Catalog scalar function. Best for column-level transformations. Access is governed by Unity Catalog permissions.
• uc-udtf: Wraps a Unity Catalog table-valued function. Best for table-level transforms like ML clustering and aggregation. Access is governed by Unity Catalog permissions.

Feature	python-run-function	uc-udf	uc-udtf
Example use case	DataFrame transforms, API integrations, email notifications	Column-level calculations (BMI, interest rates)	ML clustering, aggregation across rows
Input	DataFrames	Single values	Entire table, row by row
Output	DataFrames	Single value	Table (multiple rows)
Requires Unity Catalog function	No	Yes	Yes
Access governance	Workspace file permissions	Unity Catalog permissions (EXECUTE, USE SCHEMA)	Unity Catalog permissions (EXECUTE, USE SCHEMA)
Supported languages	Python only	SQL or Python in a SQL wrapper	SQL or Python in a SQL wrapper

Feature	python-run-function	uc-udf	uc-udtf
Example use case	DataFrame transforms, API integrations, email notifications	Column-level calculations (BMI, interest rates)	ML clustering, aggregation across rows
Input	DataFrames	Single values	Entire table, row by row
Output	DataFrames	Single value	Table (multiple rows)
Requires Unity Catalog function	No	Yes	Yes
Access governance	Workspace file permissions	Unity Catalog permissions (EXECUTE, USE SCHEMA)	Unity Catalog permissions (EXECUTE, USE SCHEMA)
Supported languages	Python only	SQL or Python in a SQL wrapper	SQL or Python in a SQL wrapper

How custom operators work

A shared custom operator consists of:

• Operator logic: The code that runs when the operator executes. This is an inline Python run() function for python-run-function operators, or a Unity Catalog function for uc-udf and uc-udtf operators. See Write logic for Lakeflow Designer custom operators.
• YAML configuration: Tells Lakeflow Designer how to present the operator in the UI, including the operator's name, description, input parameters, UI widgets, and ports. All operator types use the custom-operator-v0.1.0 schema. For a python-run-function operator, the YAML is a standalone workspace file that also holds the inline code. For a Unity Catalog function, the YAML is embedded in the function body. See Custom operator YAML reference.
• Registration file: An entry in .custom_operators.yaml that lets Lakeflow Designer discover the operator. See Register and deploy your operator to Lakeflow Designer.

Build a shared operator

To build a shared custom operator from start to finish:

1. Choose an operator type. See Choose a shared operator type.
2. Write and test the operator logic. See Write logic for Lakeflow Designer custom operators.
3. Describe the operator in YAML: its name, configuration properties, UI widgets, and ports. See Custom operator YAML reference.
4. Register the operator: Add it to .custom_operators.yaml. See Register and deploy your operator to Lakeflow Designer.
5. Grant permissions so other users can discover and run the operator. See Permissions.
6. Use the operator. It appears in the Operators pane in Lakeflow Designer, alongside the built-in operators.

Each tutorial walks through these steps for one complete operator.

tip

To create a shared python-run-function operator without writing it yourself, describe it to Genie Code in the side pane. Genie Code asks whether to register the operator for you or for the whole workspace, then writes the YAML file and registers it. You still need to grant permissions for other users.

Register and deploy your operator to Lakeflow Designer

For your operator to appear in Lakeflow Designer, register it in a .custom_operators.yaml file:

• Workspace level: Place the file in the root of your workspace to make the operator visible to all users.
• User level: Place the file in your user home folder (/Workspace/Users/<user-name>/.custom_operators.yaml) to make operators visible only to you.

note

Lakeflow Designer still reads registration files named .user_defined_operators.yaml, so existing registrations keep working.

Register new operators in .custom_operators.yaml. If the same folder already has a .user_defined_operators.yaml file, move its entries to .custom_operators.yaml. When both files exist in a folder, Lakeflow Designer reads only .custom_operators.yaml.

The operators: section supports file paths, Unity Catalog function references, and glob patterns. You can mix entry types:

YAML

operators:
  # File path (python-run-function operators)
  - /Workspace/Users/me/operators/my_operator.yaml
  # Glob pattern (registers all matching files)
  - /Workspace/Users/me/operators/transforms/*.yaml
  # UC function reference (uc-udf and uc-udtf operators)
  - catalog: my_catalog
    schema: my_schema
    functionName: my_function

Update or remove an operator

When you change an operator's code, refresh your custom operators to load the change. In the Operators tab of the menu, click .

• If the operator keeps the same version, refreshing loads the updated code.
• If the operator has a new version, the operator on the canvas prompts you to upgrade to it (or keep the current version) after you refresh.

To remove an operator from Lakeflow Designer, delete its entry from .custom_operators.yaml. For uc-udf and uc-udtf operators, you can also drop the underlying Unity Catalog function with DROP FUNCTION if you no longer need it.

Permissions

Permission	Purpose
Read access to .custom_operators.yaml.	Discover the operator.
Read access to the YAML file (python-run-function only).	Load the operator definition.
EXECUTE on the Unity Catalog function (operators based on Unity Catalog functions only).	Run the operator.
USE SCHEMA on the schema (operators based on Unity Catalog functions only).	Access the schema where the function is created.
Other permissions	Depending on your operator, users may require other permissions. For example, USE CONNECTION on a Unity Catalog connection for HTTP API calls.

Permission	Purpose
Read access to .custom_operators.yaml.	Discover the operator.
Read access to the YAML file (python-run-function only).	Load the operator definition.
EXECUTE on the Unity Catalog function (operators based on Unity Catalog functions only).	Run the operator.
USE SCHEMA on the schema (operators based on Unity Catalog functions only).	Access the schema where the function is created.
Other permissions	Depending on your operator, users may require other permissions. For example, USE CONNECTION on a Unity Catalog connection for HTTP API calls.

Troubleshooting

Issue	Solution
Operator doesn't appear in Lakeflow Designer.	Check that .custom_operators.yaml exists and lists your function or file path. For python-run-function operators, verify the file path and that the YAML file is accessible.
Schema validation fails.	Verify your YAML against the official schema at https://your-workspace.cloud.databricks.com/static/schemas/user-defined-operator-v0.1.0.json.
Permission denied.	For operators based on Unity Catalog functions, verify users have EXECUTE on the function and USE SCHEMA on the schema. For python-run-function operators, verify users have read access to the YAML file.
python-run-function operator fails at runtime.	Check that the run() function signature matches def run(config, inputs, spark). Verify that port names in the code match the YAML and that the return dictionary keys match output port name values.
UDTF returns wrong types.	UDTF return types must be explicit; you can't reference input column types.

Issue	Solution
Operator doesn't appear in Lakeflow Designer.	Check that .custom_operators.yaml exists and lists your function or file path. For python-run-function operators, verify the file path and that the YAML file is accessible.
Schema validation fails.	Verify your YAML against the official schema at https://your-workspace.cloud.databricks.com/static/schemas/user-defined-operator-v0.1.0.json.
Permission denied.	For operators based on Unity Catalog functions, verify users have EXECUTE on the function and USE SCHEMA on the schema. For python-run-function operators, verify users have read access to the YAML file.
python-run-function operator fails at runtime.	Check that the run() function signature matches def run(config, inputs, spark). Verify that port names in the code match the YAML and that the return dictionary keys match output port name values.
UDTF returns wrong types.	UDTF return types must be explicit; you can't reference input column types.

Additional resources

Explore the following tutorials:

Example	Type	Description
Gmail email sender	python-run-function	Send DataFrame data as a CSV email attachment through Gmail.
Compound interest calculator	uc-udf	Calculate future investment values using the compound interest formula.
K-means clustering	uc-udtf	Segment data into clusters using scikit-learn.
Send Slack message	uc-udf	Send notifications to Slack channels through the Slack API.
All UI widgets	uc-udf	Reference operator showcasing all available UI widgets.

Example	Type	Description
Gmail email sender	python-run-function	Send DataFrame data as a CSV email attachment through Gmail.
Compound interest calculator	uc-udf	Calculate future investment values using the compound interest formula.
K-means clustering	uc-udtf	Segment data into clusters using scikit-learn.
Send Slack message	uc-udf	Send notifications to Slack channels through the Slack API.
All UI widgets	uc-udf	Reference operator showcasing all available UI widgets.

To write the operator logic, see Write logic for Lakeflow Designer custom operators. For a complete reference to the YAML schema, see Custom operator YAML reference.