This article describes best practices when using Delta Lake.
If you expect a column to be commonly used in query predicates and if that column has high cardinality
(that is, a large number of distinct values), then use
Z-ORDER BY. Delta Lake automatically lays out
the data in the files based on the column values and use the layout information to skip irrelevant data while querying.
For details, see Z-Ordering (multi-dimensional clustering).
You can partition a Delta table by a column. The most commonly used partition column is
Follow these two rules of thumb for deciding on what column to partition by:
- If the cardinality of a column will be very high, do not use that column for partitioning. For example, if you partition by a column
userIdand if there can be 1M distinct user IDs, then that is a bad partitioning strategy.
- Amount of data in each partition: You can partition by a column if you expect data in that partition to be at least 1 GB.
If you continuously write data to a Delta table, it will over time accumulate a large number of files, especially if you add data in small batches. This can have an adverse effect on the efficiency of table reads, and it can also affect the performance of your file system. Ideally, a large number of small files should be rewritten into a smaller number of larger files on a regular basis. This is known as compaction.
You can compact a table using the OPTIMIZE command.
Sometimes you may want to replace a Delta table. For example:
- You discover the data in the table is incorrect and want to replace the content.
- You want to rewrite the whole table to do incompatible schema changes (drop columns or change column types).
While you can delete the entire directory of a Delta table and create a new table on the same path, it’s not recommended because:
- Deleting a directory is not efficient. A directory containing very large files can take hours or even days to delete.
- You lose all of content in the deleted files; it’s hard to recover if you delete the wrong table.
- The directory deletion is not atomic. While you are deleting the table a concurrent query reading the table can fail or see a partial table.
- You may experience potential consistency issues on S3, which is only eventually consistent.
If you want to change the table schema, you can replace the whole table atomically. For example:
dataframe.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .partitionBy(<your-partition-columns>) \ .saveAsTable("<your-table>") # Managed table dataframe.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .option("path", "<your-table-path>") \ .partitionBy(<your-partition-columns>) \ .saveAsTable("<your-table>") # External table
REPLACE TABLE <your-table> USING DELTA PARTITIONED BY (<your-partition-columns>) AS SELECT ... -- Managed table REPLACE TABLE <your-table> USING DELTA PARTITIONED BY (<your-partition-columns>) LOCATION "<your-table-path>" AS SELECT ... -- External table
dataframe.write .format("delta") .mode("overwrite") .option("overwriteSchema", "true") .partitionBy(<your-partition-columns>) .saveAsTable("<your-table>") // Managed table dataframe.write .format("delta") .mode("overwrite") .option("overwriteSchema", "true") .option("path", "<your-table-path>") .partitionBy(<your-partition-columns>) .saveAsTable("<your-table>") // External table
There are multiple benefits with this approach:
- Overwriting a table is much faster because it doesn’t need to list the directory recursively or delete any files.
- The old version of the table still exists. If you delete the wrong table you can easily retrieve the old data using Time Travel.
- It’s an atomic operation. Concurrent queries can still read the table while you are deleting the table.
- Because of Delta Lake ACID transaction guarantees, if overwriting the table fails, the table will be in its previous state.
- You won’t face any consistency issues on S3 because you don’t delete files.
In addition, if you want to delete old files to save storage cost after overwriting the table, you can use VACUUM to delete them. It’s optimized for file deletion and usually faster than deleting the entire directory.
Databricks recommends that you use Spark caching, that is
df = spark.read.load('/some/path') df.cache()
only if you have some expensive aggregation or join result that will be used multiple times, for example, to perform more rollups.
Otherwise, do not use this method on Delta tables.
- You lose any data skipping that can come from additional filters added on top of the cached
- The data that gets cached may not be updated if the table is accessed using a different identifier (for example, you do
spark.table(x).cache()but then write to the table using