Skip to main content

External Metadata

View as Markdown

Metadata about an object that lives outside of <Databricks>, such as a Kafka topic, a Tableau dashboard, or a Salesforce table. An external metadata object is a Unity Catalog securable that can be the source or the target of external lineage relationships, so that data flows into and out of <Databricks> appear in the Unity Catalog lineage graph. See External lineage.

ExternalMetadata object​

Metadata about an object that lives outside of <Databricks>, such as a Kafka topic, a Tableau dashboard, or a Salesforce table. An external metadata object is a Unity Catalog securable that can be the source or the target of external lineage relationships, so that data flows into and out of <Databricks> appear in the Unity Catalog lineage graph. See External lineage.

namestringPublic Preview

Name of the external metadata object. The name is unique within the metastore, must follow the Unity Catalog naming requirements, and is stored in lowercase. It cannot be changed after creation, and external lineage relationships refer to the object by this name.

Example: security_events_stream

system_typestringPublic Preview

Type of the external system that holds the object. Use OTHER for a system that is not in the list.

Values:

Show 23 possible values
  • OTHER
  • TABLEAU
  • POWER_BI
  • LOOKER
  • KAFKA
  • SAP
  • ORACLE
  • SALESFORCE
  • WORKDAY
  • MYSQL
  • POSTGRESQL
  • MICROSOFT_SQL_SERVER
  • SERVICENOW
  • AMAZON_REDSHIFT
  • AZURE_SYNAPSE
  • SNOWFLAKE
  • GOOGLE_BIGQUERY
  • MICROSOFT_FABRIC
  • MONGODB
  • TERADATA
  • CONFLUENT
  • DATABRICKS
  • STREAM_NATIVE
entity_typestringPublic Preview

Type of the object within the external system, such as a table, a report, or a dashboard. The value is free-form text of at most 255 characters.

Example: Topic

urlstringPublic Preview

URL of the object in the external system. When set, lineage graph viewers can follow it to the external asset. The URL must not exceed 2048 characters.

Example: https://kafka.com/12345

descriptionstringPublic Preview

User-provided free-form text description of the object. The description must not exceed 5000 characters.

Example: A stream of security related events in the critical services.

columnsarray of stringPublic Preview

Names of the columns of the object, used for column-level mappings in external lineage relationships. Column names must be unique within the list, and the combined size of all column names is limited.

propertiesobjectPublic Preview

Key-value properties attached to the object. Keys and values must be non-empty, and the combined length of all keys and values must not exceed 5000 characters.

ownerstringPublic Preview

Owner of the external metadata object. Defaults to the creator. To change the owner, send an update whose update mask includes owner.

Example: data-team@example.com

metastore_idstringPublic PreviewOutput only

Unique identifier of parent metastore.

create_timestringPublic PreviewOutput only

Time at which this external metadata object was created.

created_bystringPublic PreviewOutput only

Username of external metadata object creator.

update_timestringPublic PreviewOutput only

Time at which this external metadata object was last modified.

updated_bystringPublic PreviewOutput only

Username of user who last modified external metadata object.

idstringPublic PreviewOutput onlyIDImmutable

Unique identifier of the external metadata object. The ID is a UUID that the system assigns on creation.

Example: 12345678-abcd-4ef5-9012-3456789abcdf

Get an external metadata object Public Preview​

GET /api/2.0/lineage-tracking/external-metadata/{name}

Gets the specified external metadata object in a metastore. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the BROWSE privilege.

API scopes: unity-catalog

Parameters​

namestringRequiredpath

Name of the external metadata object to get.

Response​

Returns the ExternalMetadata object.

List external metadata objects Public Preview​

GET /api/2.0/lineage-tracking/external-metadata

Gets an array of external metadata objects in the metastore. If the caller is the metastore admin, all external metadata objects will be retrieved. Otherwise, only external metadata objects that the caller has BROWSE on will be retrieved. There is no guarantee of a specific ordering of the elements in the array.

API scopes: unity-catalog

Parameters​

page_sizeint32query

Specifies the maximum number of external metadata objects to return in a single response. The value must be less than or equal to 1000. Defaults to 1000 when omitted.

page_tokenstringquery

Opaque pagination token to go to next page based on previous query.

Response​

Returns a list of ExternalMetadata objects.

Create an external metadata object Public Preview​

POST /api/2.0/lineage-tracking/external-metadata

Creates a new external metadata object in the parent metastore if the caller is a metastore admin or has the CREATE_EXTERNAL_METADATA privilege. Grants BROWSE to all account users upon creation by default.

API scopes: unity-catalog

Request body​

namestringRequired

Name of the external metadata object. The name is unique within the metastore, must follow the Unity Catalog naming requirements, and is stored in lowercase. It cannot be changed after creation, and external lineage relationships refer to the object by this name.

Example: security_events_stream

system_typestringRequired

Type of the external system that holds the object. Use OTHER for a system that is not in the list.

Values:

Show 23 possible values
  • OTHER
  • TABLEAU
  • POWER_BI
  • LOOKER
  • KAFKA
  • SAP
  • ORACLE
  • SALESFORCE
  • WORKDAY
  • MYSQL
  • POSTGRESQL
  • MICROSOFT_SQL_SERVER
  • SERVICENOW
  • AMAZON_REDSHIFT
  • AZURE_SYNAPSE
  • SNOWFLAKE
  • GOOGLE_BIGQUERY
  • MICROSOFT_FABRIC
  • MONGODB
  • TERADATA
  • CONFLUENT
  • DATABRICKS
  • STREAM_NATIVE
entity_typestringRequired

Type of the object within the external system, such as a table, a report, or a dashboard. The value is free-form text of at most 255 characters.

Example: Topic

urlstring

URL of the object in the external system. When set, lineage graph viewers can follow it to the external asset. The URL must not exceed 2048 characters.

Example: https://kafka.com/12345

descriptionstring

User-provided free-form text description of the object. The description must not exceed 5000 characters.

Example: A stream of security related events in the critical services.

columnsarray of string

Names of the columns of the object, used for column-level mappings in external lineage relationships. Column names must be unique within the list, and the combined size of all column names is limited.

propertiesobject

Key-value properties attached to the object. Keys and values must be non-empty, and the combined length of all keys and values must not exceed 5000 characters.

ownerstring

Owner of the external metadata object. Defaults to the creator. To change the owner, send an update whose update mask includes owner.

Example: data-team@example.com

Response​

Returns the ExternalMetadata object.

Update an external metadata object Public Preview​

PATCH /api/2.0/lineage-tracking/external-metadata/{name}

Updates the external metadata object that matches the supplied name. The caller can only update either the owner or other metadata fields in one request. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the MODIFY privilege. If the caller is updating the owner, they must also have the MANAGE privilege.

API scopes: unity-catalog

Parameters​

namestringRequiredpath

Name of the external metadata object. The name is unique within the metastore, must follow the Unity Catalog naming requirements, and is stored in lowercase. It cannot be changed after creation, and external lineage relationships refer to the object by this name.

Example: security_events_stream

update_maskstringRequiredquery

Comma-separated list of the fields to update, for example description,properties. Only the listed fields change. The other fields keep their current values. The mask must not be empty.

Request body​

system_typestringRequired

Type of the external system that holds the object. Use OTHER for a system that is not in the list.

Values:

Show 23 possible values
  • OTHER
  • TABLEAU
  • POWER_BI
  • LOOKER
  • KAFKA
  • SAP
  • ORACLE
  • SALESFORCE
  • WORKDAY
  • MYSQL
  • POSTGRESQL
  • MICROSOFT_SQL_SERVER
  • SERVICENOW
  • AMAZON_REDSHIFT
  • AZURE_SYNAPSE
  • SNOWFLAKE
  • GOOGLE_BIGQUERY
  • MICROSOFT_FABRIC
  • MONGODB
  • TERADATA
  • CONFLUENT
  • DATABRICKS
  • STREAM_NATIVE
entity_typestringRequired

Type of the object within the external system, such as a table, a report, or a dashboard. The value is free-form text of at most 255 characters.

Example: Topic

urlstring

URL of the object in the external system. When set, lineage graph viewers can follow it to the external asset. The URL must not exceed 2048 characters.

Example: https://kafka.com/12345

descriptionstring

User-provided free-form text description of the object. The description must not exceed 5000 characters.

Example: A stream of security related events in the critical services.

columnsarray of string

Names of the columns of the object, used for column-level mappings in external lineage relationships. Column names must be unique within the list, and the combined size of all column names is limited.

propertiesobject

Key-value properties attached to the object. Keys and values must be non-empty, and the combined length of all keys and values must not exceed 5000 characters.

ownerstring

Owner of the external metadata object. Defaults to the creator. To change the owner, send an update whose update mask includes owner.

Example: data-team@example.com

Response​

Returns the ExternalMetadata object.

Delete an external metadata object Public Preview​

DELETE /api/2.0/lineage-tracking/external-metadata/{name}

Deletes the external metadata object that matches the supplied name. Every external lineage relationship in which the object is the source or the target is deleted with it. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the MANAGE privilege.

API scopes: unity-catalog

Parameters​

namestringRequiredpath

Name of the external metadata object to delete.