Common patterns for managed ingestion pipelines

Lakeflow Connect provides patterns and techniques to optimize your managed ingestion pipelines. Use these patterns to control which data is ingested, manage pipeline updates, and configure advanced behaviors.

Not all connectors support the common patterns in this section.

Topic	Description
Column selection	Select or exclude specific columns during ingestion to reduce data volume and improve performance.
Full refresh	Force a complete reload of your data from the source system.
History tracking	Track historical changes in your data using Slowly Changing Dimension (SCD) type 2.
Multi-destination pipelines	Ingest data from a single source to multiple destination tables or catalogs.
Pipeline maintenance	Manage pipeline updates, pauses, and troubleshooting workflows.
Row filtering	Filter rows during ingestion using SQL-like conditions.
Name destination tables	Name destination tables. By default, a destination table is given the name of the corresponding source table. However, naming a destination table is useful when you ingest the same source object twice in the same schema. Managed connectors don't support duplicate table names in the same destination schema. Naming a destination table can also align tables to your organization’s naming conventions.