External Metadata

View as Markdown

ExternalMetadata object

namestringPublic Preview: Name of the external metadata object.
Example: security_events_stream

system_typestringPublic Preview: Type of external system.
Values: SYSTEM_TYPE_UNSPECIFIED, OTHER, TABLEAU, POWER_BI, LOOKER, KAFKA, SAP, ORACLE, SALESFORCE, WORKDAY, MYSQL, POSTGRESQL, MICROSOFT_SQL_SERVER, SERVICENOW, AMAZON_REDSHIFT, AZURE_SYNAPSE, SNOWFLAKE, GOOGLE_BIGQUERY, MICROSOFT_FABRIC, MONGODB, TERADATA, CONFLUENT, DATABRICKS, STREAM_NATIVE

entity_typestringPublic Preview: Type of entity within the external system.
Example: Topic

urlstringPublic Preview: URL associated with the external metadata object.
Example: https://kafka.com/12345

descriptionstringPublic Preview: User-provided free-form text description.
Example: A stream of security related events in the critical services.

columnsarray of stringPublic Preview: List of columns associated with the external metadata object.

propertiesobjectPublic Preview: A map of key-value properties attached to the external metadata object.

ownerstringPublic Preview: Owner of the external metadata object.
Example: data-team@company.com

metastore_idstringPublic Preview: Unique identifier of parent metastore.

create_timestringPublic Preview: Time at which this external metadata object was created.

created_bystringPublic Preview: Username of external metadata object creator.

update_timestringPublic Preview: Time at which this external metadata object was last modified.

updated_bystringPublic Preview: Username of user who last modified external metadata object.

idstringPublic Preview: Unique identifier of the external metadata object.
Example: 12345678-abcd-4ef5-9012-3456789abcdf

View as Markdown

The ExternalMetadata object

{
  "name": "security_events_stream",
  "system_type": "string",
  "entity_type": "Topic",
  "url": "https://kafka.com/12345",
  "description": "A stream of security related events in the critical services.",
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time"
  ],
  "properties": {
    "compression.enabled": "true",
    "compression.format": "zstd",
    "topic": "prod.security.events.raw"
  },
  "owner": "data-team@company.com",
  "metastore_id": "string",
  "create_time": "string",
  "created_by": "string",
  "update_time": "string",
  "updated_by": "string",
  "id": "12345678-abcd-4ef5-9012-3456789abcdf"
}

Get Public Preview

GET /api/2.0/lineage-tracking/external-metadata/{name}

Gets the specified external metadata object in a metastore. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the BROWSE privilege.

API scopes: unity-catalog

Parameters

namestringpath

Response

Returns the ExternalMetadata object.

View as Markdown

Response

{
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time"
  ],
  "create_time": 1715802661000,
  "created_by": "john.doe@company.com",
  "description": "A stream of security related events in the critical services.",
  "entity_type": "Topic",
  "id": "12345678-abcd-4ef5-9012-3456789abcdf",
  "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
  "name": "security_events_stream",
  "owner": "data-team@company.com",
  "properties": {
    "compression.enabled": "true",
    "compression.format": "zstd",
    "topic": "prod.security.events.raw"
  },
  "system_type": "KAFKA",
  "update_time": 1715802661000,
  "updated_by": "john.doe@company.com",
  "url": "https://kafka.com/12345"
}

List Public Preview

GET /api/2.0/lineage-tracking/external-metadata

Gets an array of external metadata objects in the metastore. If the caller is the metastore admin, all external metadata objects will be retrieved. Otherwise, only external metadata objects that the caller has BROWSE on will be retrieved. There is no guarantee of a specific ordering of the elements in the array.

API scopes: unity-catalog

Parameters

page_sizeint32query: Specifies the maximum number of external metadata objects to return in a single response. The value must be less than or equal to 1000.

page_tokenstringquery: Opaque pagination token to go to next page based on previous query.

Response

Returns a list of ExternalMetadata objects.

View as Markdown

Response

{
  "external_metadata": [
    {
      "columns": [
        "type",
        "message",
        "details",
        "date",
        "time"
      ],
      "create_time": 1715802661000,
      "created_by": "john.doe@company.com",
      "description": "A stream of security related events in the critical services.",
      "entity_type": "Topic",
      "id": "12345678-abcd-4ef5-9012-3456789abcdf",
      "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
      "name": "security_events_stream",
      "owner": "data-team@company.com",
      "properties": {
        "compression.enabled": "true",
        "compression.format": "zstd",
        "topic": "prod.security.events.raw"
      },
      "system_type": "KAFKA",
      "update_time": 1715802661000,
      "updated_by": "john.doe@company.com",
      "url": "https://kafka.com/12345"
    },
    {
      "columns": [
        "customer_id",
        "age_group",
        "location",
        "segment"
      ],
      "create_time": 1715889061000,
      "created_by": "alice.johnson@company.com",
      "description": "Customer analysis report with demographics",
      "entity_type": "Report",
      "id": "87654321-dcba-4ef5-9012-3456789abcde",
      "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
      "name": "customer_report",
      "owner": "analytics-team@company.com",
      "properties": {
        "data_source": "warehouse",
        "refresh_enabled": "false"
      },
      "system_type": "TABLEAU",
      "update_time": 1716148261000,
      "updated_by": "bob.wilson@company.com",
      "url": "https://tableau.company.com/reports/customer-analysis"
    },
    {
      "columns": [
        "product_id",
        "quantity",
        "location",
        "last_updated"
      ],
      "create_time": 1716234661000,
      "created_by": "charlie.brown@company.com",
      "description": "Inventory tracking model for warehouse management",
      "entity_type": "Model",
      "id": "abcdef12-3456-7890-abcd-ef1234567890",
      "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
      "name": "inventory_model",
      "owner": "operations-team@company.com",
      "properties": {
        "model_type": "dimensional",
        "refresh_enabled": "true",
        "refresh_frequency": "real-time"
      },
      "system_type": "LOOKER",
      "update_time": 1716234661000,
      "updated_by": "charlie.brown@company.com",
      "url": "https://looker.company.com/models/inventory"
    }
  ],
  "next_page_token": "eyJwYWdlX251bWJlciI6Mn0="
}

Create Public Preview

POST /api/2.0/lineage-tracking/external-metadata

Creates a new external metadata object in the parent metastore if the caller is a metastore admin or has the CREATE_EXTERNAL_METADATA privilege. Grants BROWSE to all account users upon creation by default.

API scopes: unity-catalog

Request body

external_metadataobject

Show child attributesHide child attributes

namestring: Name of the external metadata object.
Example: security_events_stream

system_typestring: Type of external system.
Values: SYSTEM_TYPE_UNSPECIFIED, OTHER, TABLEAU, POWER_BI, LOOKER, KAFKA, SAP, ORACLE, SALESFORCE, WORKDAY, MYSQL, POSTGRESQL, MICROSOFT_SQL_SERVER, SERVICENOW, AMAZON_REDSHIFT, AZURE_SYNAPSE, SNOWFLAKE, GOOGLE_BIGQUERY, MICROSOFT_FABRIC, MONGODB, TERADATA, CONFLUENT, DATABRICKS, STREAM_NATIVE

entity_typestring: Type of entity within the external system.
Example: Topic

urlstring: URL associated with the external metadata object.
Example: https://kafka.com/12345

descriptionstring: User-provided free-form text description.
Example: A stream of security related events in the critical services.

columnsarray of string: List of columns associated with the external metadata object.

propertiesobject: A map of key-value properties attached to the external metadata object.

ownerstring: Owner of the external metadata object.
Example: data-team@company.com

metastore_idstring: Unique identifier of parent metastore.

create_timestring: Time at which this external metadata object was created.

created_bystring: Username of external metadata object creator.

update_timestring: Time at which this external metadata object was last modified.

updated_bystring: Username of user who last modified external metadata object.

idstring: Unique identifier of the external metadata object.
Example: 12345678-abcd-4ef5-9012-3456789abcdf

Response

Returns the ExternalMetadata object.

View as Markdown

Request

{
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time"
  ],
  "description": "A stream of security related events in the critical services.",
  "entity_type": "Topic",
  "name": "security_events_stream",
  "properties": {
    "compression.enabled": "true",
    "compression.format": "zstd",
    "topic": "prod.security.events.raw"
  },
  "system_type": "KAFKA",
  "url": "https://kafka.com/12345"
}

Response

{
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time"
  ],
  "create_time": 1715802661000,
  "created_by": "john.doe@company.com",
  "description": "A stream of security related events in the critical services.",
  "entity_type": "Topic",
  "id": "12345678-abcd-4ef5-9012-3456789abcdf",
  "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
  "name": "security_events_stream",
  "owner": "data-team@company.com",
  "properties": {
    "compression.enabled": "true",
    "compression.format": "zstd",
    "topic": "prod.security.events.raw"
  },
  "system_type": "KAFKA",
  "update_time": 1715802661000,
  "updated_by": "john.doe@company.com",
  "url": "https://kafka.com/12345"
}

Update Public Preview

PATCH /api/2.0/lineage-tracking/external-metadata/{external_metadata.name}

Updates the external metadata object that matches the supplied name. The caller can only update either the owner or other metadata fields in one request. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the MODIFY privilege. If the caller is updating the owner, they must also have the MANAGE privilege.

API scopes: unity-catalog

Parameters

external_metadataobjectpath

Show child attributesHide child attributes

namestring: Name of the external metadata object.
Example: security_events_stream

system_typestring: Type of external system.
Values: SYSTEM_TYPE_UNSPECIFIED, OTHER, TABLEAU, POWER_BI, LOOKER, KAFKA, SAP, ORACLE, SALESFORCE, WORKDAY, MYSQL, POSTGRESQL, MICROSOFT_SQL_SERVER, SERVICENOW, AMAZON_REDSHIFT, AZURE_SYNAPSE, SNOWFLAKE, GOOGLE_BIGQUERY, MICROSOFT_FABRIC, MONGODB, TERADATA, CONFLUENT, DATABRICKS, STREAM_NATIVE

entity_typestring: Type of entity within the external system.
Example: Topic

urlstring: URL associated with the external metadata object.
Example: https://kafka.com/12345

descriptionstring: User-provided free-form text description.
Example: A stream of security related events in the critical services.

columnsarray of string: List of columns associated with the external metadata object.

propertiesobject: A map of key-value properties attached to the external metadata object.

ownerstring: Owner of the external metadata object.
Example: data-team@company.com

metastore_idstring: Unique identifier of parent metastore.

create_timestring: Time at which this external metadata object was created.

created_bystring: Username of external metadata object creator.

update_timestring: Time at which this external metadata object was last modified.

updated_bystring: Username of user who last modified external metadata object.

idstring: Unique identifier of the external metadata object.
Example: 12345678-abcd-4ef5-9012-3456789abcdf

update_maskstringquery

Response

Returns the ExternalMetadata object.

View as Markdown

Request

{
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time",
    "severity"
  ],
  "description": "Updated stream of security related events in the critical services.",
  "entity_type": "Topic",
  "name": "security_events_stream",
  "properties": {
    "compression.enabled": "true",
    "compression.format": "lz4",
    "topic": "prod.security.events.raw"
  },
  "system_type": "KAFKA",
  "update_mask": "url,description,columns,properties",
  "url": "https://kafka.com/12345-updated"
}

Response

{
  "columns": [
    "type",
    "message",
    "details",
    "date",
    "time",
    "severity"
  ],
  "create_time": 1715802661000,
  "created_by": "john.doe@company.com",
  "description": "Updated stream of security related events in the critical services.",
  "entity_type": "Topic",
  "id": "12345678-abcd-4ef5-9012-3456789abcdf",
  "metastore_id": "12345678-abcd-4ef5-9012-3456789abcde",
  "name": "security_events_stream",
  "owner": "data-team@company.com",
  "properties": {
    "compression.enabled": "true",
    "compression.format": "lz4",
    "topic": "prod.security.events.raw"
  },
  "system_type": "KAFKA",
  "update_time": 1717394661000,
  "updated_by": "jane.smith@company.com",
  "url": "https://kafka.com/12345-updated"
}

Delete Public Preview

DELETE /api/2.0/lineage-tracking/external-metadata/{name}

Deletes the external metadata object that matches the supplied name. The caller must be a metastore admin, the owner of the external metadata object, or a user that has the MANAGE privilege.

API scopes: unity-catalog

Parameters

namestringpath