Online Table

View as Markdown

Online Table information.

OnlineTable object

Online Table information.

namestring: Full three-part (catalog, schema, table) name of the table.

specobject

Specification of the online table.

Show child attributesHide child attributes

run_continuouslyobject: Pipeline runs continuously after generating the initial data.

run_triggeredobject: Pipeline stops after generating the initial data and can be triggered later (manually, through a cron job or through data triggers)

source_table_full_namestring: Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of string: Primary Key columns to be used for data insert/update in the destination.

timeseries_keystring: Time series key to deduplicate (tie-break) rows with the same primary key.

perform_full_copyboolean: Whether to create a full-copy pipeline -- a pipeline that stops after creates a full copy of the source table upon initialization and does not process any change data feeds (CDFs) afterwards. The pipeline can still be manually triggered afterwards, but it always perform a full copy of the source table and there are no incremental updates. This mode is useful for syncing views or tables without CDFs to online tables. Note that the full-copy pipeline only supports "triggered" scheduling policy.
Default: false

pipeline_idstring: ID of the associated pipeline. Generated by the server - cannot be set by the caller.

statusobject

Online Table data synchronization status

Show child attributesHide child attributes

detailed_statestring: The state of the online table.
Values: ONLINE_TABLE_STATE_UNSPECIFIED, PROVISIONING, PROVISIONING_PIPELINE_RESOURCES, PROVISIONING_INITIAL_SNAPSHOT, ONLINE, ONLINE_CONTINUOUS_UPDATE, ONLINE_TRIGGERED_UPDATE, ONLINE_NO_PENDING_UPDATE, OFFLINE, OFFLINE_FAILED, ONLINE_PIPELINE_FAILED, ONLINE_UPDATING_PIPELINE_RESOURCES

messagestring: A text description of the current state of the online table.

provisioning_statusobject

Show child attributesHide child attributes

initial_pipeline_sync_progressobject

Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.

Show child attributesHide child attributes

latest_version_currently_processingint64: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64: The number of rows that have been synced in this update.

total_row_countint64: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondouble: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdouble: The estimated time remaining to complete this update in seconds.

continuous_update_statusobject

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may not be completely synced to the online table yet.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table.

initial_pipeline_sync_progressobject

Progress of the initial data synchronization.

Show child attributesHide child attributes

latest_version_currently_processingint64: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64: The number of rows that have been synced in this update.

total_row_countint64: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondouble: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdouble: The estimated time remaining to complete this update in seconds.

triggered_update_statusobject

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may not be completely synced to the online table yet.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table.

triggered_update_progressobject

Progress of the active data synchronization pipeline.

Show child attributesHide child attributes

latest_version_currently_processingint64: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64: The number of rows that have been synced in this update.

total_row_countint64: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondouble: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdouble: The estimated time remaining to complete this update in seconds.

failed_statusobject

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may only be partially synced to the online table. Only populated if the table is still online and available for serving.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table. Only populated if the table is still online and available for serving.

table_serving_urlstring: Data serving REST API URL for this table

unity_catalog_provisioning_statestring: The provisioning state of the online table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).
Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

View as Markdown

The OnlineTable object

{
  "name": "string",
  "spec": {
    "run_continuously": {},
    "run_triggered": {},
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "perform_full_copy": true,
    "pipeline_id": "string"
  },
  "status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {}
  },
  "table_serving_url": "string",
  "unity_catalog_provisioning_state": "string"
}

Get

GET /api/2.0/online-tables/{name}

Get information about an existing online table and its status.

API scopes: unity-catalog

Parameters

namestringpath: Full three-part (catalog, schema, table) name of the table.

Response

Returns the OnlineTable object.

View as Markdown

Response

{
  "name": "string",
  "spec": {
    "run_continuously": {},
    "run_triggered": {},
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "perform_full_copy": true,
    "pipeline_id": "string"
  },
  "status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {}
  },
  "table_serving_url": "string",
  "unity_catalog_provisioning_state": "string"
}

Create

POST /api/2.0/online-tables

Create a new Online Table.

API scopes: unity-catalog

Request body

tableobject

Specification of the online table to be created.

Show child attributesHide child attributes

namestring: Full three-part (catalog, schema, table) name of the table.

specobject

Specification of the online table.

Show child attributesHide child attributes

run_continuouslyobjectRequired: Pipeline runs continuously after generating the initial data.

run_triggeredobjectRequired: Pipeline stops after generating the initial data and can be triggered later (manually, through a cron job or through data triggers)

source_table_full_namestring: Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of string: Primary Key columns to be used for data insert/update in the destination.

timeseries_keystring: Time series key to deduplicate (tie-break) rows with the same primary key.

perform_full_copyboolean: Whether to create a full-copy pipeline -- a pipeline that stops after creates a full copy of the source table upon initialization and does not process any change data feeds (CDFs) afterwards. The pipeline can still be manually triggered afterwards, but it always perform a full copy of the source table and there are no incremental updates. This mode is useful for syncing views or tables without CDFs to online tables. Note that the full-copy pipeline only supports "triggered" scheduling policy.
Default: false

pipeline_idstring: ID of the associated pipeline. Generated by the server - cannot be set by the caller.

statusobject

Online Table data synchronization status

Show child attributesHide child attributes

detailed_statestring: The state of the online table.
Values: ONLINE_TABLE_STATE_UNSPECIFIED, PROVISIONING, PROVISIONING_PIPELINE_RESOURCES, PROVISIONING_INITIAL_SNAPSHOT, ONLINE, ONLINE_CONTINUOUS_UPDATE, ONLINE_TRIGGERED_UPDATE, ONLINE_NO_PENDING_UPDATE, OFFLINE, OFFLINE_FAILED, ONLINE_PIPELINE_FAILED, ONLINE_UPDATING_PIPELINE_RESOURCES

messagestring: A text description of the current state of the online table.

provisioning_statusobjectRequired

Show child attributesHide child attributes

initial_pipeline_sync_progressobject: Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.

continuous_update_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may not be completely synced to the online table yet.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table.

initial_pipeline_sync_progressobject: Progress of the initial data synchronization.

triggered_update_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may not be completely synced to the online table yet.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table.

triggered_update_progressobject: Progress of the active data synchronization pipeline.

failed_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was synced to the online table. Note that this Delta version may only be partially synced to the online table. Only populated if the table is still online and available for serving.

timestampstring: The timestamp of the last time any data was synchronized from the source table to the online table. Only populated if the table is still online and available for serving.

table_serving_urlstring: Data serving REST API URL for this table

unity_catalog_provisioning_statestring: The provisioning state of the online table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).
Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

Response

Returns the OnlineTable object.

View as Markdown

Request

{
  "table": {
    "name": "string",
    "spec": {},
    "status": {},
    "table_serving_url": "string",
    "unity_catalog_provisioning_state": "string"
  }
}

Response

{
  "name": "string",
  "spec": {
    "run_continuously": {},
    "run_triggered": {},
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "perform_full_copy": true,
    "pipeline_id": "string"
  },
  "status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {}
  },
  "table_serving_url": "string",
  "unity_catalog_provisioning_state": "string"
}

Delete

DELETE /api/2.0/online-tables/{name}

Delete an online table. Warning: This will delete all the data in the online table. If the source Delta table was deleted or modified since this Online Table was created, this will lose the data forever!

API scopes: unity-catalog

Parameters

namestringpath: Full three-part (catalog, schema, table) name of the table.

OnlineTable object​

Get​

Parameters​

Response​

Create​

Request body​

Response​

Delete​

Parameters​

OnlineTable object

Get

Parameters

Response

Create

Request body

Response

Delete

Parameters