Sinks in Lakeflow Spark Declarative Pipelines

By default when you create a flow, your pipeline writes the resulting query to a Delta table, typically a materialized view or a streaming table. Pipelines also provide functionality to let you write to a wide range of sinks, or even programmatically transform and stream data to any target (or targets) that you can write to with Python.

The following topics describe the sink functionality in pipelines.

Topic	Description
Lakeflow Spark Declarative Pipelines sinks	Use the sink API with flows to write records transformed by a pipeline to a supported external data sink. External data sinks include Unity Catalog managed and external tables, and event streaming services such as Apache Kafka or Azure Event Hubs.
Python custom sinks	Use the sink API with a Python custom data source to write to an arbitrary data store.
ForEachBatch sinks	Use the foreachBatch API to write to an arbitrary data store and perform other tranformations on the data or write to multiple sinks within a single flow.

More information

• Lakeflow Spark Declarative Pipelines
• Flows