Delta Live Tables frequently asked questions

Can I query views in a pipeline from another cluster or SQL warehouse?

No, currently views are only available to other tables in the pipeline.

Does Delta Live Tables only support updating of Delta tables?

Yes, Delta Live Tables can only be used to update Delta tables.

Does Delta Live Tables perform maintenance tasks on my tables?

Yes, Delta Live Tables performs maintenance tasks on tables every 24 hours. Maintenance can improve query performance and reduce cost by removing old versions of tables. By default, the system performs a full OPTIMIZE operation followed by VACUUM. You can disable OPTIMIZE for a table by setting pipelines.autoOptimize.managed = false in the table properties for the table. Maintenance tasks are performed only if a pipeline update has run in the 24 hours before the maintenance tasks are scheduled.

To ensure the maintenance cluster has the required storage location access, you must apply security configurations required to access your storage locations to both the default cluster and the maintenance cluster. See Cluster configuration.

Can I query an older snapshot of a table?

By default, Delta tables created by a Delta Live Tables pipeline retain seven days of history, allowing you to query a snapshot of a table. You can modify the retention period using table properties if you require more than seven days of history.

Can I have multiple queries in a pipeline writing to the same target table?

No, each table must be defined once. You can use UNION If you need to combine multiple inputs to create a table. Adding or removing UNION from a streaming live table is a breaking operation that requires a full-refresh.

Can I use Scala or Java libraries in a Delta Live Tables pipeline?

No, Delta Live Tables supports only SQL and Python. You cannot use JVM libraries in a pipeline. Installing JVM libraries will cause unpredictable behavior, and may break with future Delta Live Tables releases.

How do I use identity columns when creating live tables with SQL?

Identity columns can be specified in the schema definition of a CREATE LIVE TABLE statement in the same way as the standard CREATE TABLE syntax. However, for live tables that are not defined as STREAMING, each refresh may completely regenerate the table, resulting in a new assignment of identity values to each row.

Identity columns are not supported with APPLY CHANGES tables.