Publish data from Delta Live Tables pipelines

You can make the output data of your pipeline discoverable and available to query by publishing datasets to the Databricks metastore. To publish datasets to the metastore, enter a database name in the Target field when you create a pipeline. You can also add a target database to an existing pipeline:

1. Click the Settings button.

2. Add the target setting to configure a database name for your tables.

   

3. Click Save.

4. Click the button to start a new update for your pipeline.

After the update completes, you can view the database and tables, query the data, or use the data in downstream applications.

You can use this feature with multiple environment configurations to publish to different databases based on the environment. For example, you can publish to a dev database for development and a prod database for production data.

When you create a target configuration, only tables and associated metadata are published. Views are not published to the metastore.

Exclude tables

To prevent publishing of intermediate tables that are not intended for external consumption, mark them as TEMPORARY:

CREATE TEMPORARY LIVE TABLE temp_table
AS SELECT ... ;

@dlt.table(
  temporary=True)
def temp_table():
  return ("...")