Skip to main content
Last updated on

Access materialized views and streaming tables using external systems

By default, materialized views and streaming tables are not available to access from external systems. Databricks provides two features that allow you to make your datasets available to Delta Lake or Iceberg clients:

  • External data access (Public Preview) - Enabling external data access for pipeline datasets creates a copy of the metadata for the dataset that allows clients to use the Unity Catalog or Iceberg REST APIs, without requiring a full data copy or separate refresh schedule. The clients must use the catalog REST APIs, and must support either Delta 4.0.0 or above, or the Iceberg v3 specification.

    For details about external data access for pipeline datasets, see Enable external data access to streaming tables and materialized views.

  • Compatibility mode (Public Preview) - Enabling compatibility mode for a dataset generates a read-only version of your data at a chosen location, which must be updated when the tables are refreshed. The compatibility version includes v1 metadata for both Delta Lake and Iceberg formats along with the read-only copy of the data, allowing access for a wider range of clients (including clients that must read the table data directly, rather than through an API) at the expense of delay on data updates, and data copying costs.

    For details about compatibility mode, see Compatibility Mode.

If your external clients support the REST APIs, Databricks recommends using external data access to give those clients access to materialized views or streaming tables. For a wider range of clients, especially older clients, and other Unity Catalog managed tables, use compatibility mode.

Capability

External data access

Compatibility mode

Data copy

No data copy required.

Data copy required.

Consistency

Read-after-write consistency. External clients see changes as soon as they happen in the dataset.

Updates happen on a schedule. By default, the compatibility table updates hourly. This can be set to update immediately after a change to the source table, but it is still delayed by data-copy time.

Access

Requires "modern" REST API access. Supports Delta 4.0.0 or above catalog APIs or Iceberg v3 specification APIs (requires support of deletion vectors).

Compatible with all Delta Lake or Iceberg clients.

Single table object

Materialized views and streaming tables appear as managed tables with the same name as the original dataset to the external clients.

Compatibility tables appear as a new table in a new location to the external clients.

Table type support

Supports materialized views and streaming tables managed by Lakeflow Spark Declarative Pipelines.

Supports materialized views and streaming tables, whether managed by Lakeflow Spark Declarative Pipelines or standalone, and any other Unity Catalog managed tables.

Cost

The cost of maintaining the external-facing metadata is part of the refresh cost for the materialized view or streaming table. This is generally under 1% of the cost and time for the refresh.

The majority of the cost for compatibility mode is the cost of transferring the legacy data to the new location.

Next steps​

On this page