Skip to main content

Uc External Lineage

View as Markdown

A lineage relationship between a source object and a target object, where at least one of the two is an external metadata object. The other object can be a Unity Catalog table, a cloud storage path, a model version, or another external metadata object. A relationship is identified by its source and target; optional column mappings and key-value properties describe the data flow between them.

ExternalLineageRelationship object​

A lineage relationship between a source object and a target object, where at least one of the two is an external metadata object. The other object can be a Unity Catalog table, a cloud storage path, a model version, or another external metadata object. A relationship is identified by its source and target; optional column mappings and key-value properties describe the data flow between them.

idstringPublic PreviewOutput onlyIDImmutable

Unique identifier of the external lineage relationship.

Example: 12345678-abcd-4ef5-9012-3456789abcdf

sourceobjectPublic Preview

Source object of the external lineage relationship.

Show child attributesHide child attributes
tableobjectPublic Preview

A Unity Catalog table.

Show child attributesHide child attributes
namestringPublic Preview

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobjectPublic Preview

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstringPublic Preview

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobjectPublic Preview

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestringPublic Preview

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstringPublic Preview

Version number of the model, as a string.

Example: 1

external_metadataobjectPublic Preview

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestringPublic Preview

Name of the external metadata object.

Example: example_crm_accounts

targetobjectPublic Preview

Target object of the external lineage relationship.

Show child attributesHide child attributes
tableobjectPublic Preview

A Unity Catalog table.

Show child attributesHide child attributes
namestringPublic Preview

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobjectPublic Preview

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstringPublic Preview

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobjectPublic Preview

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestringPublic Preview

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstringPublic Preview

Version number of the model, as a string.

Example: 1

external_metadataobjectPublic Preview

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestringPublic Preview

Name of the external metadata object.

Example: example_crm_accounts

columnsarray of objectPublic Preview

List of column relationships between source and target objects. The combined size of all source and target column names is limited.

Show child attributesHide child attributes
sourcestringPublic Preview

Name of the column in the source object.

targetstringPublic Preview

Name of the column in the target object.

propertiesobjectPublic Preview

Key-value properties associated with the external lineage relationship. The combined length of all keys and values must not exceed 5000 characters.

List external lineage relationships Public Preview​

GET /api/2.0/lineage-tracking/external-lineage

Lists external lineage relationships of a <Databricks> object or external metadata given a supplied direction.

API scopes: unity-catalog

Parameters​

object_infoobjectRequiredquery

The object to query external lineage relationships for. Since this field is a query parameter, please flatten the nested fields. For example, if the object is a table, the query parameter should look like: object_info.table.name=main.sales.customers

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

lineage_directionstringRequiredquery

Direction of the relationships to return, relative to object_info. UPSTREAM returns the relationships in which the object is the target, so each result describes a source; DOWNSTREAM returns the relationships in which the object is the source, so each result describes a target.

Values:

  • UPSTREAM
  • DOWNSTREAM
page_sizeint32query

Specifies the maximum number of external lineage relationships to return in a single response. The value must be less than or equal to 1000.

page_tokenstringquery

Opaque pagination token to go to next page based on previous query.

Response​

external_lineage_relationshipsarray of object

Relationships that match the query. Each entry describes the object at the other end of the relationship in one of table_info, file_info, model_info, or external_metadata_info, together with the relationship itself in external_lineage_info.

Show child attributesHide child attributes
table_infoobject

Information about the table involved in the lineage relationship.

Show child attributesHide child attributes
namestring

Name of Table.

catalog_namestring

Name of Catalog.

schema_namestring

Name of Schema.

event_timestring

Timestamp of the lineage event.

file_infoobject

Information about the file involved in the lineage relationship.

Show child attributesHide child attributes
pathstring

URL of the path.

securable_namestring

The full name of the securable on the path.

storage_locationstring

The storage location associated with securable on the path.

securable_typestring

The securable type of the securable on the path.

event_timestring

Timestamp of the lineage event.

model_infoobject

Information about the model version involved in the lineage relationship.

Show child attributesHide child attributes
model_namestring

Name of the model.

versionint64

Version number of the model.

event_timestring

Timestamp of the lineage event.

external_metadata_infoobject

Information about external metadata involved in the lineage relationship.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

system_typestring

Type of external system.

Values:

Show 23 possible values
  • OTHER
  • TABLEAU
  • POWER_BI
  • LOOKER
  • KAFKA
  • SAP
  • ORACLE
  • SALESFORCE
  • WORKDAY
  • MYSQL
  • POSTGRESQL
  • MICROSOFT_SQL_SERVER
  • SERVICENOW
  • AMAZON_REDSHIFT
  • AZURE_SYNAPSE
  • SNOWFLAKE
  • GOOGLE_BIGQUERY
  • MICROSOFT_FABRIC
  • MONGODB
  • TERADATA
  • CONFLUENT
  • DATABRICKS
  • STREAM_NATIVE
entity_typestring

Type of entity represented by the external metadata object.

event_timestring

Timestamp of the lineage event.

external_lineage_infoobject

Information about the edge metadata of the external lineage relationship.

Show child attributesHide child attributes
idstringOutput onlyIDImmutable

Unique identifier of the external lineage relationship.

Example: 12345678-abcd-4ef5-9012-3456789abcdf

sourceobjectRequired

Source object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

targetobjectRequired

Target object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

columnsarray of object

List of column relationships between source and target objects. The combined size of all source and target column names is limited.

Show child attributesHide child attributes
sourcestring

Name of the column in the source object.

targetstring

Name of the column in the target object.

propertiesobject

Key-value properties associated with the external lineage relationship. The combined length of all keys and values must not exceed 5000 characters.

next_page_tokenstring

Opaque token to retrieve the next page of results. Omitted when there are no more results.

Create an external lineage relationship Public Preview​

POST /api/2.0/lineage-tracking/external-lineage

Creates an external lineage relationship between a <Databricks> or external metadata object and another external metadata object.

API scopes: unity-catalog

Request body​

sourceobjectRequired

Source object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

targetobjectRequired

Target object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

columnsarray of object

List of column relationships between source and target objects. The combined size of all source and target column names is limited.

Show child attributesHide child attributes
sourcestring

Name of the column in the source object.

targetstring

Name of the column in the target object.

propertiesobject

Key-value properties associated with the external lineage relationship. The combined length of all keys and values must not exceed 5000 characters.

Response​

Returns the ExternalLineageRelationship object.

Update an external lineage relationship Public Preview​

PATCH /api/2.0/lineage-tracking/external-lineage

Updates an external lineage relationship between a <Databricks> or external metadata object and another external metadata object.

API scopes: unity-catalog

Parameters​

update_maskstringRequiredquery

The fields of the relationship to update. The field mask must be a single string, with multiple fields separated by commas (no spaces), for example columns,properties. The source and target identify the relationship and cannot be changed.

Request body​

sourceobjectRequired

Source object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

targetobjectRequired

Target object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

columnsarray of object

List of column relationships between source and target objects. The combined size of all source and target column names is limited.

Show child attributesHide child attributes
sourcestring

Name of the column in the source object.

targetstring

Name of the column in the target object.

propertiesobject

Key-value properties associated with the external lineage relationship. The combined length of all keys and values must not exceed 5000 characters.

Response​

Returns the ExternalLineageRelationship object.

Delete an external lineage relationship Public Preview​

DELETE /api/2.0/lineage-tracking/external-lineage

Deletes an external lineage relationship between a <Databricks> or external metadata object and another external metadata object.

API scopes: unity-catalog

Parameters​

external_lineage_relationshipobjectRequiredquery

The relationship to delete, identified by its source and target objects. Since this field is a query parameter, flatten the nested fields, for example external_lineage_relationship.source.table.name=main.sales.orders.

Show child attributesHide child attributes
sourceobjectRequired

Source object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

targetobjectRequired

Target object of the external lineage relationship.

Show child attributesHide child attributes
tableobject

A Unity Catalog table.

Show child attributesHide child attributes
namestring

Full name of the table, in the form catalog.schema.table.

Example: main.sales.orders

pathobject

A cloud storage path registered in Unity Catalog, i.e. under a table, volume, or external location.

Show child attributesHide child attributes
urlstring

URL of the cloud storage path. The path must be registered in Unity Catalog, i.e. under a table, volume, or external location.

Example: s3://my-bucket/landing/orders

model_versionobject

A version of a Unity Catalog registered model.

Show child attributesHide child attributes
namestring

Full name of the registered model, in the form catalog.schema.model.

Example: main.ml.churn_model

versionstring

Version number of the model, as a string.

Example: 1

external_metadataobject

An external metadata object registered in Unity Catalog.

Show child attributesHide child attributes
namestring

Name of the external metadata object.

Example: example_crm_accounts

columnsarray of object

List of column relationships between source and target objects. The combined size of all source and target column names is limited.

Show child attributesHide child attributes
sourcestring

Name of the column in the source object.

targetstring

Name of the column in the target object.

propertiesobject

Key-value properties associated with the external lineage relationship. The combined length of all keys and values must not exceed 5000 characters.