Skip to main content

Delta Lake limitations on S3

Delta Lake on Databricks has specific considerations when storing tables in Amazon S3, including bucket versioning, concurrent writes, and risks from deleting files directly.

Bucket versioning and Delta Lake​

Databricks recommends that you don't turn on bucket versioning for buckets that store Delta Lake data, including Unity Catalog managed tables. Delta Lake implements its own versioning and garbage collection.

When you turn on bucket versioning, S3 retains copies of metadata and data files that manual and automated processes on Databricks consider deleted. This includes data files that VACUUM would permanently delete and transaction logs cleaned up automatically during regular Delta Lake table operations.

important

If you choose to use bucket versioning, Databricks recommends retaining three versions and implementing a lifecycle management policy that retains versions for 7 days or less for all S3 buckets with versioning enabled.

If you encounter performance slowdown on tables stored in buckets with versioning enabled, mention that bucket versioning is enabled when contacting Databricks support.

Concurrent writes to Delta Lake tables on S3​

Databricks automatically selects the write mechanism for Delta Lake transaction log commits to S3. You don't need to configure the write mechanism.

Databricks supports concurrent writes from multiple clusters and workspaces. For classic compute that has been running since before June 9, 2026, restart the compute before writing concurrently.

Custom and external writers must use atomic S3 conditional writes or an equivalent commit mechanism compatible with every writer. Concurrent writes are unsafe if any writer lacks atomic commit coordination.

Deprecated

Setting spark.databricks.delta.multiClusterWrites.enabled to false is deprecated and unsupported. Leaving this setting configured can block operations or make concurrent writes unsafe.

Remove this setting from compute Spark configurations, compute policies, job definitions, notebook code, and deployment templates. Restart the compute after you remove the setting.

Resolve a blocked multi-cluster writes configuration​

You might receive the following error when a compute configuration overrides the deprecated setting:

Text
This operation has been blocked because spark.databricks.delta.multiClusterWrites.enabled=false was found to be overridden in this session. Using this configuration is unsafe for concurrent writes. Remove this override to access this table.

To resolve this error:

  1. Remove spark.databricks.delta.multiClusterWrites.enabled from the compute configuration, compute policy, job definition, notebook code, or deployment template.
  2. Restart the compute and retry the operation.

If the operation remains blocked after the compute restarts, contact Databricks support.

Resolve a blocked cross-cloud write​

Databricks blocks writes to S3 from compute outside AWS when the write can't provide transactional guarantees.

If a cross-cloud write is blocked, run the workload on AWS compute or contact Databricks support.

Stale data after deleting files with rm -rf​

Don't use rm -rf to drop a Delta Lake table. See Drop or replace a table.