Automatic upgrades
Automatic upgrades for existing schemas are in Public Preview. To enroll, complete this form with your account ID. No code changes or additional configuration are required after enrollment.
Automatic upgrades for new schemas are generally available.
For Unity Catalog managed tables, Databricks automatically upgrades them to use generally available recommended features without requiring code changes or manual ALTER TABLE statements. Automatic upgrades also verify that clients are compatible before turning on new features.
Automatic upgrades provide the following benefits:
- Reduce the administrative effort necessary to validate individual compatibility requirements for each table and feature combination in your workspaces, especially if you have catalogs with thousands of tables.
- Automatically gain the latest performance and reliability improvements on your managed tables.
- Implement upgrades safely. Automatic upgrades only turn on features after they verify workload compatibility.
How automatic upgrades work
Automatic upgrades monitor Unity Catalog access patterns at both the table and schema levels and use an observation window to verify that access patterns are compatible before turning on any feature. The observation window is 50 days for features in Public Preview, and 100 days for generally available features.
Automatic upgrades use serverless compute to upgrade tables in the background.
Schemas and tables
The behavior of automatic upgrades depends on whether schemas and tables existed before automatic upgrades were turned on. This table has more detail:
Schema | Table | Behavior |
|---|---|---|
New | New | Automatic upgrades set schema-level defaults at creation time so that tables immediately inherit all supported features without an observation period. |
Existing | New | Automatic upgrades turn on a feature if only verified workloads have accessed the table within the observation window. Otherwise, if a single unverified workload accessed the table, automatic upgrades ignore the table. See verified workloads. |
Existing | Existing | Automatic upgrades turn on a feature when all of the following are true:
|
Verified workloads
A workload is considered verified for a specific feature if it accessed the table from a Databricks cluster with a Databricks Runtime version at or above the feature's minimum required version.
The following workloads are considered unverified:
- External clients and third-party services such as Flink or Presto. See Unity Catalog integrations.
- Databricks services with direct or kernel-level table access, such as Zerobus, that bypass standard Databricks Runtime access patterns. See Zerobus Ingest connector overview.
If any table in a schema was accessed within the observation window by a Databricks Runtime version below the feature's minimum required version or by an external client, automatic upgrades don't turn on the corresponding feature on any table in that schema.
Supported features
Automatic upgrades may automatically turn on the following features:
Feature | What it does | Release status | Minimum compatible Databricks Runtime version |
|---|---|---|---|
Maintains hidden row IDs for incremental processing with change data feed. | Generally available for new tables in new schemas. Public preview for all tables in existing schemas. | 14.1 | |
Allows Delta Lake to support more concurrent writers and reduces write conflicts on large or frequently updated tables. | Generally available for new tables in new schemas. Public preview for all tables in existing schemas. | 13.3 | |
Centralizes commits in Unity Catalog to allow multi-table transactions, improve interoperability for external writes, and allow governance policies across engines. | Public preview for all tables in all schemas. | 16.4 | |
Automatically organizes table data based on frequently queried columns to improve query performance without manual partitioning. | Generally available for new tables in new schemas. Automatic upgrades for this feature aren't allowed for all tables in existing schemas. | 13.3 |
Feature availability might differ by region.
Requirements
- Serverless compute must be available in your region.
- Tables must be Unity Catalog managed tables in Delta Lake or Apache Iceberg formats.
Observe enabled features
To check if automatic upgrades turned on a feature on a table, look for a SET TBLPROPERTIES operation in the History tab in Catalog Explorer, or use DESCRIBE HISTORY <table_name>. If automatic upgrades performed the operation, the username field shows a hash value instead of a username, such as 4d137f29-62. See What is Catalog Explorer? and View table history.
After automatic upgrades turn on features on tables in a new schema, view the schema defaults in the Properties tab in Catalog Explorer. For example, a schema with row tracking turned on shows a property such as catalog.schema.enableRowTracking: "true". Existing schemas don't have automatic upgrades observability properties.
Manage recommended features
Administrators can manage the behavior and operations of upgrades with various controls.
Revert changes
Use RESTORE to revert the data and metadata for a table to a version before the feature was turned on:
RESTORE TABLE <table_name> TO VERSION AS OF <version>;
RESTORE TABLE <table_name> TO TIMESTAMP AS OF <timestamp>;
See Restore a table to an earlier state for more information about table history and restore.
Turn off features for tables
To turn off a feature on an individual table:
ALTER TABLE <table_name> DROP FEATURE <feature_name>
Automatic upgrades don't turn on a feature again after you manually turn it off.
Limitations
- Tables shared by Delta Lake Sharing, both Databricks-to-Open and Databricks-to-Databricks, are excluded from automatic upgrades. See What is Delta Sharing?.
- Automatic upgrades do not have a batch rollback mechanism to turn off a feature across all tables in an account. See Manage automatic upgrades recommended features.
- Materialized views and streaming tables are not supported.
- Workloads that bypass Unity Catalog and access tables directly by file path are not tracked by automatic upgrades. If your workloads use path-based access, contact your account team to discuss compatibility.
- External tables are typically accessed by file path, bypassing Unity Catalog, and with unverified workloads from external clients. Unity Catalog cannot reliably track these access patterns, so external tables are excluded from automatic upgrades. See Work with external tables.