Skip to main content

Move DLT tables between pipelines

note

This feature is in Public Preview and only available to participating customers at this time. If you would like to participate in this feature preview, contact your Databricks account manager to enable it for your workspace.

This article describes how to move streaming tables and materialized views between pipelines. The new pipeline will update the table, rather than the source table. This is useful in many scenarios, including:

  • Split a large pipeline into smaller ones.
  • Merge multiple pipelines in a single larger one.
  • Change the refresh frequency for some tables in a pipeline.
  • Move tables from the legacy LIVE schema, to the new default publishing mode. See LIVE schema (legacy).

Requirements​

The following are prerequisites for moving a table between pipelines.

  • You must use Databricks Runtime 16.3 or above when running the ALTER TABLE command needed.
  • Both source and destination pipelines must be:
    • Unity Catalog pipelines (Hive metastore pipelines are not supported)
    • In the same workspace
    • Owned by the Databricks user account or service principal running the operation
  • You must run the destination pipeline at least once before moving the table.
  • You must have configured the destination pipeline to use the default publishing mode. This enables you to publish tables to multiple catalogs and schemas. Otherwise, both pipelines must be using the legacy publishing mode and both must have the same catalog and target value in settings.
note

This feature does not support moving a pipeline using the default publishing mode to a pipeline using the legacy publishing mode.

Move a table between pipelines​

The following instructions describe how to move a streaming table or materialized view from one pipeline to another.

  1. Stop the source pipeline if it is running. Wait for it to completely stop.

  2. Remove the table’s definition from the source pipeline’s notebook or file and store it somewhere for future reference.

    Include any supporting queries or code that is needed for the pipeline to run correctly.

  3. From a notebook or a SQL editor, run the following SQL command to reassign the table from the source pipeline to the destination pipeline:

    SQL
    ALTER [MATERIALIZED VIEW | STREAMING TABLE] <table-name>
      SET TBLPROPERTIES("pipelines.pipelineId"="<destination-pipeline-id>");

    For example, if you want to move a streaming table named sales to a pipeline with the ID abcd1234-ef56-ab78-cd90-1234efab5678, you would run the following command:

    SQL
    ALTER STREAMING TABLE sales
      SET TBLPROPERTIES("pipelines.pipelineId"="abcd1234-ef56-ab78-cd90-1234efab5678");

  4. Add the table’s definition to the destination pipeline’s notebook/file.

    note

    If the catalog or target schema differ between the source and destination, copying the query exactly might not work. Partially qualified tables in the definition can resolve differently. You might need to update the definition while moving.

The move is complete. You can now run both the source and destination pipelines. The destination pipeline udpates the table.

Troubleshooting​

The following table describes errors that could happen when moving a table between pipelines.

Error

Description

DESTINATION_PIPELINE_NOT_IN_DIRECT_PUBLISHING_MODE

The source pipeline is in the default publish mode, and the destination uses the LIVE schema (legacy) mode. This is not supported. If the source uses the default publishing mode, then the destination must, as well.

CHANGE_PIPELINE_ID_OF_TABLE_NOT_SUPPORTED

This feature is in a public preview, and might not be enabled for your workspace. Contact your Databricks account manager to enable it for your workspace.

PIPELINE_TYPE_NOT_WORKSPACE_PIPELINE_TYPE

Only moving tables between ETL pipelines is supoprted. Pipelines for streaming tables and materialized views created with Databricks SQL are not supported.

Table fails to update in the destination after the move.

To quickly mitigate in this case, move the table back to the source pipeline following the same instructions. You can then troubleshoot or raise the issue with your account manager.

Limitations​

The following are limitations for moving tables between pipelines.

  • Materialized views and streaming tables created with Databricks SQL are not supported.
  • Private tables or views are not supported.
  • The source and destination pipelines must be Unity Catalog managed pipelines. Hive metastore pipelines are not supported.
  • Both source and destination pipelines must be in the same workspace.
  • Both source and destination pipelines must be owned by the user running the move operation.
  • If the source pipeline uses the default publishing mode, the destination pipeline must also be using the default publishing mode. You can't move a table from a pipeline using the default publishing mode to a pipeline that uses the LIVE schema (legacy). See LIVE schema (legacy).
  • If the source and destination pipelines are both using the LIVE schema (legacy), then they must have the same catalog and target values in settings.
Last updated on