Skip to main content

Use Apache Iceberg v3 features

Beta

This feature is in Beta.

This page describes how to use Apache Iceberg v3 features with Unity Catalog. Iceberg v3 enhances query performance and introduces new features for managed Delta Lake tables with UniForm, managed Iceberg tables, and foreign Iceberg tables.

The key features of Iceberg v3 are:

  • Deletion vectors: Enable efficient, row-level deletes without rewriting entire data files.
  • VARIANT data type: Supports storing and processing semi-structured data.
  • Row lineage: Tracks incremental changes to table data.

Row lineage is required for all Iceberg v3 tables. There's no command to enable or disable row lineage.

Requirements​

To use Iceberg v3 features, you must meet the following requirements:

  • A workspace with Unity Catalog enabled
  • Databricks Runtime 17.3 or above to read and write to managed tables with Iceberg v3

Create a new table with Iceberg v3​

Create new tables with Iceberg v3 enabled for both managed Delta tables with UniForm and managed Iceberg tables.

To create a new managed Delta table with UniForm and Iceberg v3 enabled, use the following SQL command:

SQL
CREATE OR REPLACE TABLE main.schema.table (c1 INT) TBLPROPERTIES(
  'delta.universalFormat.enabledFormats' = 'iceberg',
  'delta.enableIcebergCompatV3' = 'true'
);

For more information about UniForm, see Read Delta tables with Iceberg clients.

Upgrade an existing table to Iceberg v3​

You can upgrade an existing table to Iceberg v3 by:

  1. Enabling any v3 feature on a table.
  2. Setting the Iceberg format version on a table to 3 (shown below).
warning

Managed Iceberg tables that are upgraded to v3 can't be downgraded. Managed Delta tables with UniForm can be safely downgraded.

To upgrade a managed Delta table with UniForm to v3, use the following command:

SQL
ALTER TABLE catalog.schema.table SET TBLPROPERTIES(
  'delta.enableIcebergCompatV3' = 'true',
  'delta.enableIcebergCompatV2' = 'false'
);

Enable deletion vectors​

Deletion vectors optimize row-level data modification operations and are enabled by default on all new Iceberg v3 tables. See What are deletion vectors?.

note

Enabling deletion vectors on an existing Iceberg table upgrades the Iceberg format version to 3.

To create a new managed Delta table with UniForm, Iceberg v3, and deletion vectors enabled, set the following table properties:

SQL
CREATE TABLE catalog.schema.table (c1 INT) TBLPROPERTIES(
  'delta.enableDeletionVectors' = 'true',
  'delta.enableIcebergCompatV3' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Use the VARIANT data type​

The VARIANT data type allows you to store and query semi-structured data.

note

Using VARIANT in an existing Iceberg table upgrades the Iceberg format version to 3.

To create a new managed Delta table with UniForm and a VARIANT column:

SQL
CREATE TABLE catalog.schema.deltaTable (col VARIANT) TBLPROPERTIES(
  'delta.enableIcebergCompatV3' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

To add a VARIANT column to an existing table, use the ALTER TABLE command:

SQL
ALTER TABLE catalog.schema.table ADD COLUMN variant_col VARIANT;

Restore a table to a previous version​

If you need to revert a table to a state before it was upgraded to Iceberg v3, you can use the RESTORE command.

SQL
RESTORE TABLE catalog.schema.table TO VERSION AS OF 1;

Limitations​

Databricks supports version 3 of the Iceberg specification, with the following exceptions:

  • Defaults, including write defaults and initial defaults, aren't supported.
  • The following data types aren't supported:
    • Geospatial types
    • Unknown type
    • Nanosecond precision timestamp.
  • Multi-argument transforms aren't supported.
Last updated on