Delta Lake limitations on S3
This article details some of the limitations you might encounter while working with data stored in S3 with Delta Lake on Databricks. The eventually consistent model used in Amazon S3 can lead to potential problems when multiple systems or clusters modify data in the same table simultaneously.
Databricks and Delta Lake support multi-cluster writes by default, meaning that queries writing to a table from multiple clusters at the same time won’t corrupt the table. For Delta tables stored on S3, this guarantee is limited to a single Databricks workspace.
To avoid potential data corruption and data loss issues, Databricks recommends you do not modify the same Delta table stored in S3 from different workspaces.
Bucket versioning and Delta Lake
Databricks recommends against enabling bucket versioning for buckets that store Delta Lake data, including Unity Catalog managed tables. Delta Lake implements its own versioning and garbage collection.
When you enable bucket versioning, S3 retains copies of metadata and data files that manual and automated processes on Databricks consider deleted. This includes data files that VACUUM would permanently delete and transaction logs cleaned up automatically during regular Delta table operations.
If you choose to use bucket versioning, Databricks recommends retaining three versions and implementing a lifecycle management policy that retains versions for 7 days or less for all S3 buckets with versioning enabled.
If you encounter performance slowdown on tables stored in buckets with versioning enabled, please indicate that bucket versioning is enabled while contacting Databricks support.
What are the limitations of multi-cluster writes on S3?
The following features are not supported when running in this mode:
- Server-Side Encryption with Customer-Provided Encryption Keys
- S3 paths with credentials in a cluster that cannot access AWS Security Token Service
You can disable multi-cluster writes by setting spark.databricks.delta.multiClusterWrites.enabled to false. If they are disabled, writes to a single table must originate from a single cluster.
Disabling spark.databricks.delta.multiClusterWrites.enabled and modifying the same Delta table from multiple clusters concurrently can lead to data loss or data corruption.
Why does a table show old data after I delete Delta Lake files with rm -rf and create a new table in the same location?
You should never use rm -rf to drop a Delta table. See Drop or replace a Delta table.