Common patterns for managed ingestion pipelines

Lakeflow Connect provides patterns and techniques to optimize your managed ingestion pipelines. Use these patterns to control which data is ingested, manage pipeline updates, and configure advanced behaviors.

Not all connectors support the common patterns in this section.

Topic	Description
Column selection	Select or exclude specific columns during ingestion to reduce data volume and improve performance.
Full refresh	Force a complete reload of your data from the source system.
History tracking	Track historical changes in your data using Slowly Changing Dimension (SCD) type 2.
Multi-destination	Ingest data from a single source to multiple destination tables or catalogs.
Pipeline maintenance	Manage pipeline updates, pauses, and troubleshooting workflows.
Table renaming	Rename destination tables without recreating your pipeline.