Skip to main content
Last updated on

Catalog commits

Catalog commits shift commit coordination from the file system to Unity Catalog, making the catalog the single source of truth for Delta Lake and Apache Iceberg table state. This enables multi-table transactions, faster query planning, and governed access across all table operations.

Traditional Delta Lake transactions coordinate commits at the individual table level. Each table manages its own transaction log and conflict detection independently. By moving commit coordination to the catalog level, this allows organizations to consistently govern all access to the Lakehouse through Unity Catalog. It also allows Unity Catalog to orchestrate commits across multiple tables within a single transaction boundary while maintaining Delta Lake's ACID guarantees.

Benefits​

Catalog commits provide the following capabilities:

  • Transactions spanning multiple tables: Run multiple SQL statements across multiple tables as a single atomic commit. All changes succeed together or fail together. See Transactions.

    Preview

    Transactions that write to Unity Catalog managed Iceberg tables are in Private Preview. To join this preview, submit the managed Iceberg tables preview enrollment form.

  • Governed access: Reads and writes are coordinated through Unity Catalog, so engines see the latest committed state and governance policies are applied.

  • Faster query planning and writes: Unity Catalog informs a Delta client of table-level metadata directly when it accesses a table, skipping cloud storage and removing a major source of metadata latency.

  • Enforceable constraints: Unity Catalog validates or rejects schema and constraint changes, preventing incompatible updates that could break data integrity or downstream workloads.

  • External access: Safely write to Unity Catalog managed tables from external engines. Unity Catalog coordinates commits to prevent corruption and concurrency conflicts. See Access Databricks data using external systems.

    Beta

    This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Requirements​

  • Tables must be Unity Catalog managed tables (Delta or Iceberg), including streaming tables, or materialized views.
  • Databricks Runtime 16.4 and above is required to read from, write to, or create managed tables with catalog commits enabled.
  • Databricks Runtime 18.0 and above is required to enable or disable catalog commits on existing managed tables.
  • Databricks Runtime 17.3 and above is required to read from or write to streaming tables or materialized views with catalog commits enabled.
  • In Spark Declarative Pipelines, serverless compute or classic compute with Databricks Runtime 17.3 and above is required to read from, write to, create, or upgrade streaming tables or materialized views with catalog commits enabled.
  • Databricks Runtime 18 LTS and above is required to disable catalog commits on existing streaming tables or materialized views.

Enable catalog commits​

Enable catalog commits for new or existing managed tables, streaming tables, and materialized views.

Enable catalog commits for new managed tables​

Use the delta.feature.catalogManaged table property when creating a managed table:

SQL
CREATE TABLE sales_data (
  sale_id BIGINT,
  amount DECIMAL(10,2),
  sale_date DATE
)
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');

Enable catalog commits for existing managed tables​

Use ALTER TABLE to add catalog commits to an existing managed table:

SQL
ALTER TABLE sales_data SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');
important

Enabling catalog commits on an existing table synchronizes the table state with the catalog. This operation can take several minutes on tables with high volumes of write operations.

Enable catalog commits for new or existing streaming tables or materialized views​

Add the delta.feature.catalogManaged table property to the streaming table or materialized view definition, and then run an update. For an existing object, update its original definition. ALTER TABLE ... SET TBLPROPERTIES is not supported.

The following example enables catalog commits for a new or existing streaming table:

SQL
CREATE OR REFRESH STREAMING TABLE streaming_sales_data
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported')
AS SELECT * FROM STREAM sales_data;

For a materialized view, use the same table property in its CREATE OR REFRESH MATERIALIZED VIEW definition.

Check if catalog commits are enabled​

To verify whether a table has catalog commits enabled:

SQL
DESCRIBE DETAIL sales_data;

If enabled, catalogManaged appears in the tableFeatures column.

Turn off catalog commits​

Use Databricks Runtime 18.0 and above to turn off catalog commits on an existing managed table.

For an existing streaming table or materialized view, use Databricks Runtime 18 LTS and above.

See Drop a Delta Lake table feature and downgrade table protocol.

warning

Do not cancel upgrade or downgrade operations while executing ALTER or DROP statements. Interruption might leave the table in a partially upgraded or downgraded state, locking it from all future reads and writes. To revert, run the appropriate command again rather than canceling. Contact Databricks support if the table becomes locked.

Limitations​

  • You cannot turn on or turn off catalog commits on existing managed tables using CREATE OR REPLACE TABLE or REPLACE TABLE. Use CREATE TABLE with the delta.feature.catalogManaged property to turn it on when creating a managed table, or ALTER TABLE to turn it on or turn it off on an existing managed table.
  • Catalog commits are not compatible with external data access on streaming tables or materialized views. To use catalog commits, you must first disable external access. See Enable external data access to streaming tables and materialized views.
  • Tables with catalog commits enabled are shared through OpenSharing using pre-signed URLs instead of cloud tokens.
  • Single-user clusters cannot access streaming tables that have catalog commits enabled.

Additional resources​

On this page