Low shuffle merge on Databricks

Delta Lake updates a table by rewriting whole data files. When a MERGE command updates or deletes a few rows in a file, it must also rewrite every unmodified row in that file. Without low shuffle merge, those unmodified rows pass through the same shuffle stages as the modified rows. This makes MERGE slower and scrambles the existing data layout, so later queries also slow down.

Low shuffle merge is the optimized Databricks implementation of MERGE. It processes unmodified rows separately from modified rows, without shuffles or other expensive calculations. As a result, MERGE shuffles much less data and runs faster.

Low shuffle merge is enabled by default, so you don't need to change your MERGE statements to benefit from it. It helps most when a MERGE updates a relatively small number of rows in a large table.

Data layout after a merge

The earlier MERGE implementation changed the data layout of unmodified data entirely, which slowed down later operations. Low shuffle merge tries to preserve the existing layout of unmodified rows, including liquid clustering layout. As a result, performance degrades more slowly after one or more MERGE commands, and you need to run OPTIMIZE less often.

The layout of updated or newly inserted rows might not be optimal. You might still need to run OPTIMIZE on tables with liquid clustering enabled.

For tables that use Z-ordering, low shuffle merge also tries to preserve the existing Z-order layout of unmodified rows. You might still need to run OPTIMIZE ZORDER BY after a MERGE operation. Databricks recommends using liquid clustering for all new tables.