Use Delta Live Tables pipelines with legacy Hive metastore
This article details configurations and caveats specific to Delta Live Tables pipelines configured to publish data to the legacy Hive metastore. Databricks recommends using Unity Catalog for all new pipelines. See Use Unity Catalog with your Delta Live Tables pipelines.
Note
This article discusses functionality for the current default publishing mode for pipelines. Pipelines created before February 5, 2025, might use the legacy publishing mode and LIVE
virtual schema. See LIVE schema (legacy).
How to query streaming tables and materialized views in the legacy Hive metastore
After an update is complete, you can view the schema and tables, query the data, or use the data in downstream applications.
Once published, Delta Live Tables tables can be queried from any environment with access to the target schema. This includes Databricks SQL, notebooks, and other Delta Live Tables pipelines.
Important
When you create a target
configuration, only tables and associated metadata are published. Views are not published to the metastore.
Configure a pipeline to publish to Hive metastore
When creating a new pipeline, you can specify Hive metastore under the Storage options to publish to the legacy Hive metastore. You must specify a default target schema when publishing to Hive metastore. See Configure a Delta Live Tables pipeline.
Specify a storage location
You can specify a storage location for a pipeline that publishes to the Hive metastore. The primary motivation for specifying a location is to control the object storage location for data written by your pipeline. Databricks recommends always specificying a storage location to avoid writing to the DBFS root.
Because all tables, data, checkpoints, and metadata for Delta Live Tables pipelines are fully managed by Delta Live Tables, most interaction with Delta Live Tables datasets happens through tables registered to the Hive metastore or Unity Catalog.
Cloud storage configuration
You use AWS instance profiles to configure access to S3 storage in AWS. To add an instance profile in the Delta Live Tables UI when you create or edit a pipeline:
On the Pipeline details page for your pipeline, click the Settings button.
In the Instance profile drop-down menu In the Compute section of the pipeline settings, select an instance profile.
To configure an AWS instance profile by editing the JSON settings for your pipeline clusters, click the JSON button and enter the instance profile configuration in the aws_attributes.instance_profile_arn
field in the cluster configuration:
{
"clusters": [
{
"aws_attributes": {
"instance_profile_arn": "arn:aws:..."
}
}
]
}
You can also configure instance profiles when you create cluster policies for your Delta Live Tables pipelines. For an example, see the knowledge base.
Example pipeline source code notebooks for workspaces without Unity Catalog
You can import the following notebooks into a Databricks workspace without Unity Catalog enabled and use them to deploy a Delta Live Tables pipeline. Import the notebook of your chosen language and specify the path in Source code field when configuring a pipeline with the Hive metastore storage option. See Configure a Delta Live Tables pipeline.