Skip to main content

Use partial schema inference in Python Lakeflow pipelines

Partial schema inference lets you define selected columns in a Python Lakeflow pipeline and infer the remaining columns from the query output. Use the override_schema argument to specify types, supported constraints, or comments for one or more columns without declaring every output column.

Beta

Partial schema inference is in Beta. In workspaces where this Beta is available, use override_schema in a Python workspace pipeline to opt in for a table.

Choose a schema option​

The following options control the output schema of a materialized view, streaming table, or managed table:

Option

Behavior

Omit both arguments

Infer the output schema from the query.

schema

Declare the complete output schema.

override_schema

Declare selected column definitions and infer the remaining query output columns.

Option

Behavior

Omit both arguments

Infer the output schema from the query.

schema

Declare the complete output schema.

override_schema

Declare selected column definitions and infer the remaining query output columns.

Both schema and override_schema accept a SQL DDL string or a PySpark StructType. Specify only one of these arguments. A DDL string in override_schema defines columns inside the Python API. It does not enable partial schema declarations in SQL statements.

Use override_schema with table, materialized_view, or create_streaming_table. For managed tables, use create_table where that API is available. The corresponding functions in the dlt module also accept this argument. For new code, use pyspark.pipelines.

Define selected columns​

  1. Add override_schema to the decorator or target-table definition.
  2. Declare the columns whose definitions you want to control. The query types must be compatible with the declared types. Cast incompatible values in the query before writing them.
  3. Return the other columns you want to include from the query. Partial schema inference includes query output columns, not source columns that the query does not select.

The following pipeline creates a materialized view with an explicit NOT NULL constraint on id. It infers the definitions of name and score from the query:

Python
from pyspark import pipelines as dp

@dp.materialized_view(override_schema="id BIGINT NOT NULL")
def customers():
return spark.range(3).selectExpr(
"id",
"concat('customer-', CAST(id AS STRING)) AS name",
"id * 10 AS score",
)

The output contains id BIGINT NOT NULL, name STRING, and score BIGINT. Adding another selected column to the query includes that column without adding it to override_schema.

To use a StructType, declare only the selected fields. The following streaming table reads an accessible, append-only Delta Lake source table whose columns are id BIGINT, event_time TIMESTAMP, and payload STRING. The source must contain non-null id values:

Python
from pyspark import pipelines as dp
from pyspark.sql.types import LongType, StructField, StructType

@dp.table(
override_schema=StructType([StructField("id", LongType(), nullable=False)])
)
def events():
return spark.readStream.table("catalog.schema.raw_events").select(
"id", "event_time", "payload"
)

For a separately defined target, pass override_schema to dp.create_streaming_table() or dp.create_table() and use an append flow to write to it. Each non-generated column declared in override_schema must appear in the combined query output of the flows feeding the target. It can be absent from an individual append flow.

Column definitions and constraints​

You can include column comments and supported table constraints in an override_schema DDL string. Existing requirements for these features apply; primary and foreign keys remain informational.

Declare a complete complex column, such as a struct, array, or map. You cannot declare only some children of a nested struct and infer its remaining children.

Schema changes between updates​

When you change the query or partial declaration, the following rules apply to updates without a full refresh:

  • For a recomputed materialized view, inferred columns that are absent from the new query output are removed from the output schema.
  • For a streaming table, previously inferred columns remain in the table when later appends omit them. Existing values are retained, and later rows have NULL in the omitted columns.
  • Removing a column from override_schema also removes its declared NOT NULL requirement. If the query still returns that column, its nullability is inferred.

Review downstream queries before changing an output schema. A new declaration does not validate every historical file or automatically repair previously committed data.

Limitations​

This Beta does not support partial schema declarations in SQL or for materialized views and streaming tables created outside a workspace pipeline, such as in Databricks SQL.

Partial schema inference does not support the following:

  • Target tables for AUTO CDC or AUTO CDC FROM SNAPSHOT flows.
  • DEFAULT or IDENTITY column definitions in override_schema.
  • Declaring only part of a nested struct.

Prepare for schema changes​

Test declaration changes on representative data and retain the previous pipeline code. For streaming tables, confirm that the source data needed to replay an update remains available before considering a full refresh.

For an optional additional recovery option, prepare pipeline rewind before changing schemas and retain a usable pre-change rewind point from a successful update. Rewind has separate eligibility and setup requirements. Follow its retention and replay guidance when planning recovery. Partial schema inference does not require rewind.