Enrich Delta Lake tables with custom metadata
Databricks recommends always providing comments for tables and columns in tables. You can generate these comments using AI. See Add AI-generated comments to a table.
Unity Catalog also provides the ability to tag data. See Apply tags to Unity Catalog securable objects.
You can also log messages for individual commits to tables in a field in the Delta Lake transaction log.
Set user-defined commit metadata
You can specify user-defined strings as metadata in commits, either using the DataFrameWriter option userMetadata
or the SparkSession configuration spark.databricks.delta.commitInfo.userMetadata
. If both of them have been specified, then the option takes preference. This user-defined metadata is readable in the DESCRIBE HISTORY
operation. See Work with Delta Lake table history.
SET spark.databricks.delta.commitInfo.userMetadata=some-comment
INSERT OVERWRITE target_table SELECT * FROM data_source
df.write \
.mode("overwrite") \
.option("userMetadata", "some-comment") \
.table("target_table")
df.write
.mode("overwrite")
.option("userMetadata", "some-comment")
.table("target_table")