Skip to main content
Last updated on

Requirements for standalone pipelines

This page describes the requirements for creating and refreshing standalone materialized views and streaming tables.

You can create and refresh standalone materialized views and streaming tables using a SQL warehouse. To submit CREATE and REFRESH statements, use the SQL editor in the Databricks UI, the Databricks SQL CLI, or the Databricks SQL API.

General requirements​

The following requirements apply to all standalone pipelines.

You must have:

Permissions to create or refresh​

The owner (the user who creates the table) must have the following permissions:

  • SELECT privilege on the base tables.
  • USE CATALOG and USE SCHEMA privileges on the catalog and schema containing the source tables.
  • USE CATALOG and USE SCHEMA privileges on the target catalog and schema.
  • CREATE MATERIALIZED VIEW privilege on the schema containing the materialized view.
  • CREATE TABLE privilege on the schema containing the streaming table. Pipelines using legacy publishing mode also require the CREATE TABLE privilege for materialized views.

To refresh a standalone materialized view or streaming table:

  • You must be in the workspace that created it.
  • You must have the REFRESH privilege on the table. Owners have this privilege implicitly.

Source table requirements​

For incremental refresh of materialized views from Delta tables, the source tables must have row tracking enabled.

SQL warehouses​

To create or refresh standalone materialized views and streaming tables using a SQL warehouse, you must have a Unity Catalog-enabled pro or serverless SQL warehouse.

You can create and refresh standalone materialized views and streaming tables from a notebook attached to serverless general compute. This option is useful when you want to define and run materialized views or streaming tables alongside other notebook-based workflows without provisioning a SQL warehouse.

Serverless general compute requirements​

  • A notebook attached to serverless general compute.
  • Databricks Runtime 18.1 or above. Interactive notebooks meet this requirement automatically; jobs pinned to an older version do not.
  • Your workspace must be in a supported region.

Limitations​

  • Only the table owner can refresh the table. To allow another user to refresh, change the owner. See Change the owner of a streaming table and Change the owner of a materialized view.
  • Asynchronous refreshes are not supported. Use a synchronous refresh instead.
  • The preview channel is not supported. Tables created on serverless general compute use the current channel.
  • A table can only be refreshed using the compute type it was created with. A table created on a SQL warehouse must be refreshed on a SQL warehouse, and a table created on serverless general compute must be refreshed on serverless general compute. To check the compute type, view the table in Catalog Explorer.
  • Cost attribution and control are not available. Use a SQL warehouse if you need per-table cost attribution.
  • Vertical autoscaling on out-of-memory errors is not available.
  • Retries for schema upgrades are not available.
  • Performance mode selection on refresh is not available. See Select a performance mode for scheduled refreshes.
note

spark.sql is supported when running a refresh in a notebook on serverless general compute.

:::

Query requirements​

To query a standalone materialized view or streaming table, you must be the owner, or you must have SELECT on the table along with USE CATALOG and USE SCHEMA on its parents.

You must use one of the following compute resources:

  • SQL warehouse
  • Lakeflow Spark Declarative Pipelines interfaces
  • Standard access mode compute (formerly shared access mode)
  • Dedicated access mode compute (formerly single user access mode) on Databricks Runtime 15.4 or above, if the workspace is enabled for serverless compute. See Fine-grained access control on dedicated compute. If you are the owner, you can use dedicated access mode compute running Databricks Runtime 14.3 or above.

Regional availability​

Tables created and refreshed using a Databricks SQL warehouse are available in all regions that support serverless Databricks SQL warehouses.

For the list of supported regions for Databricks SQL warehouses, see Serverless availability.

On this page