Synced Database Table

SyncedDatabaseTable object

namestringPublic Preview: Full three-part (catalog, schema, table) name of the table.

database_instance_namestringPublic Preview: Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).

effective_database_instance_namestringPublic Preview: The name of the database instance that this table is registered to. This field is always returned, and for tables inside database catalogs is inferred database instance associated with the catalog. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

logical_database_namestringPublic Preview

Target Postgres database object (logical database) name for this table.

When creating a synced table in a registered Postgres catalog, the target Postgres database name is inferred to be that of the registered catalog. If this field is specified in this scenario, the Postgres database name MUST match that of the registered catalog (or the request will be rejected).

When creating a synced table in a standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. Note that this has implications for the create_database_objects_is_missing field in spec.

effective_logical_database_namestringPublic Preview: The name of the logical database that this table is registered to. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

specobjectPublic Preview

Show child attributesHide child attributes

scheduling_policystringPublic Preview: Scheduling policy of the underlying pipeline.
Values: SYNCED_TABLE_SCHEDULING_POLICY_UNSPECIFIED, CONTINUOUS, TRIGGERED, SNAPSHOT

source_table_full_namestringPublic Preview: Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of stringPublic Preview: Primary Key columns to be used for data insert/update in the destination.

timeseries_keystringPublic Preview: Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstringPublic Preview

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If existing_pipeline_id is defined, the synced table will be bin packed into the existing pipeline referenced. This avoids creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline.

create_database_objects_if_missingbooleanPublic Preview: If true, the synced table's logical database and schema resources in PG will be created if they do not already exist.
Default: true

new_pipeline_specobjectPublic Preview

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If new_pipeline_spec is defined, a new pipeline is created for this synced table. The location pointed to is used to store intermediate files (checkpoints, event logs etc). The caller must have write permissions to create Delta tables in the specified catalog and schema. Again, note this requires write permissions, whereas the source table only requires read permissions.

Show child attributesHide child attributes

storage_catalogstringPublic Preview

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastringPublic Preview

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstringBeta: Budget policy to set on the newly created pipeline.

unity_catalog_provisioning_statestringPublic Preview: The provisioning state of the synced table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).
Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

data_synchronization_statusobjectPublic Preview

Synced Table data synchronization status

Show child attributesHide child attributes

detailed_statestringPublic Preview: The state of the synced table.
Values: SYNCED_TABLE_STATE_UNSPECIFIED, SYNCED_TABLE_PROVISIONING, SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES, SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT, SYNCED_TABLE_ONLINE, SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE, SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE, SYNCED_TABLED_OFFLINE, SYNCED_TABLE_OFFLINE_FAILED, SYNCED_TABLE_ONLINE_PIPELINE_FAILED, SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES

messagestringPublic Preview: A text description of the current state of the synced table.

provisioning_statusobjectPublic Preview

Show child attributesHide child attributes

initial_pipeline_sync_progressobjectPublic Preview

Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.

Show child attributesHide child attributes

latest_version_currently_processingint64Public Preview: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public Preview: The number of rows that have been synced in this update.

total_row_countint64Public Preview: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic Preview: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic Preview: The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic Preview: The current phase of the data synchronization pipeline.
Values: PROVISIONING_PHASE_UNSPECIFIED, PROVISIONING_PHASE_MAIN, PROVISIONING_PHASE_INDEX_SCAN, PROVISIONING_PHASE_INDEX_SORT

continuous_update_statusobjectPublic Preview

Show child attributesHide child attributes

last_processed_commit_versionint64Public Preview: The last source table Delta version that was successfully synced to the synced table.

timestampstringPublic Preview: The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

initial_pipeline_sync_progressobjectPublic Preview

Progress of the initial data synchronization.

Show child attributesHide child attributes

latest_version_currently_processingint64Public Preview: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public Preview: The number of rows that have been synced in this update.

total_row_countint64Public Preview: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic Preview: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic Preview: The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic Preview: The current phase of the data synchronization pipeline.
Values: PROVISIONING_PHASE_UNSPECIFIED, PROVISIONING_PHASE_MAIN, PROVISIONING_PHASE_INDEX_SCAN, PROVISIONING_PHASE_INDEX_SORT

triggered_update_statusobjectPublic Preview

Show child attributesHide child attributes

last_processed_commit_versionint64Public Preview: The last source table Delta version that was successfully synced to the synced table.

timestampstringPublic Preview: The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

triggered_update_progressobjectPublic Preview

Progress of the active data synchronization pipeline.

Show child attributesHide child attributes

latest_version_currently_processingint64Public Preview: The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public Preview: The number of rows that have been synced in this update.

total_row_countint64Public Preview: The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic Preview: The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic Preview: The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic Preview: The current phase of the data synchronization pipeline.
Values: PROVISIONING_PHASE_UNSPECIFIED, PROVISIONING_PHASE_MAIN, PROVISIONING_PHASE_INDEX_SCAN, PROVISIONING_PHASE_INDEX_SORT

failed_statusobjectPublic Preview

Show child attributesHide child attributes

last_processed_commit_versionint64Public Preview: The last source table Delta version that was successfully synced to the synced table. The last source table Delta version that was synced to the synced table. Only populated if the table is still synced and available for serving.

timestampstringPublic Preview: The end timestamp of the last time any data was synchronized from the source table to the synced table. Only populated if the table is still synced and available for serving.

pipeline_idstringPublic Preview: ID of the associated pipeline. The pipeline ID may have been provided by the client (in the case of bin packing), or generated by the server (when creating a new pipeline).

last_syncobjectPublic Preview

Summary of the last successful synchronization from source to destination.

Will always be present if there has been a successful sync. Even if the most recent syncs have failed.

Limitation: The only exception is if the synced table is doing a FULL REFRESH, then the last sync information will not be available until the full refresh is complete. This limitation will be addressed in a future version.

This top-level field is a convenience for consumers who want easy access to last sync information without having to traverse detailed_status.

Show child attributesHide child attributes

sync_start_timestampstringPublic Preview: The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestampstringPublic Preview: The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobjectPublic Preview

Show child attributesHide child attributes

delta_commit_versionint64Public Preview: The Delta Lake commit version that was last successfully synced.

delta_commit_timestampstringPublic Preview: The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.

View as Markdown

The SyncedDatabaseTable object

{
  "name": "string",
  "database_instance_name": "string",
  "effective_database_instance_name": "string",
  "logical_database_name": "string",
  "effective_logical_database_name": "string",
  "spec": {
    "scheduling_policy": "string",
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "existing_pipeline_id": "string",
    "create_database_objects_if_missing": true,
    "new_pipeline_spec": {}
  },
  "unity_catalog_provisioning_state": "string",
  "data_synchronization_status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {},
    "pipeline_id": "string",
    "last_sync": {}
  }
}

Get Public Preview

GET /api/2.0/database/synced_tables/{name}

Get a Synced Database Table.

API scopes: postgres

Parameters

namestringpath

Response

Returns the SyncedDatabaseTable object.

View as Markdown

Response

{
  "name": "string",
  "database_instance_name": "string",
  "effective_database_instance_name": "string",
  "logical_database_name": "string",
  "effective_logical_database_name": "string",
  "spec": {
    "scheduling_policy": "string",
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "existing_pipeline_id": "string",
    "create_database_objects_if_missing": true,
    "new_pipeline_spec": {}
  },
  "unity_catalog_provisioning_state": "string",
  "data_synchronization_status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {},
    "pipeline_id": "string",
    "last_sync": {}
  }
}

Create Public Preview

POST /api/2.0/database/synced_tables

Create a Synced Database Table.

API scopes: postgres

Request body

synced_tableobject

Show child attributesHide child attributes

namestring: Full three-part (catalog, schema, table) name of the table.

database_instance_namestring: Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).

effective_database_instance_namestring: The name of the database instance that this table is registered to. This field is always returned, and for tables inside database catalogs is inferred database instance associated with the catalog. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

logical_database_namestring

Target Postgres database object (logical database) name for this table.

effective_logical_database_namestring: The name of the logical database that this table is registered to. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

specobject

Show child attributesHide child attributes

scheduling_policystring: Scheduling policy of the underlying pipeline.
Values: SYNCED_TABLE_SCHEDULING_POLICY_UNSPECIFIED, CONTINUOUS, TRIGGERED, SNAPSHOT

source_table_full_namestring: Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of string: Primary Key columns to be used for data insert/update in the destination.

timeseries_keystring: Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstring

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

create_database_objects_if_missingboolean: If true, the synced table's logical database and schema resources in PG will be created if they do not already exist.
Default: true

new_pipeline_specobject

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

Show child attributesHide child attributes

storage_catalogstring

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstringBeta: Budget policy to set on the newly created pipeline.

unity_catalog_provisioning_statestring: The provisioning state of the synced table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).
Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

data_synchronization_statusobject

Synced Table data synchronization status

Show child attributesHide child attributes

detailed_statestring: The state of the synced table.
Values: SYNCED_TABLE_STATE_UNSPECIFIED, SYNCED_TABLE_PROVISIONING, SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES, SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT, SYNCED_TABLE_ONLINE, SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE, SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE, SYNCED_TABLED_OFFLINE, SYNCED_TABLE_OFFLINE_FAILED, SYNCED_TABLE_ONLINE_PIPELINE_FAILED, SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES

messagestring: A text description of the current state of the synced table.

provisioning_statusobjectRequired

Show child attributesHide child attributes

initial_pipeline_sync_progressobject: Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.

continuous_update_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was successfully synced to the synced table.

timestampstring: The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

initial_pipeline_sync_progressobject: Progress of the initial data synchronization.

triggered_update_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was successfully synced to the synced table.

timestampstring: The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

triggered_update_progressobject: Progress of the active data synchronization pipeline.

failed_statusobjectRequired

Show child attributesHide child attributes

last_processed_commit_versionint64: The last source table Delta version that was successfully synced to the synced table. The last source table Delta version that was synced to the synced table. Only populated if the table is still synced and available for serving.

timestampstring: The end timestamp of the last time any data was synchronized from the source table to the synced table. Only populated if the table is still synced and available for serving.

pipeline_idstring: ID of the associated pipeline. The pipeline ID may have been provided by the client (in the case of bin packing), or generated by the server (when creating a new pipeline).

last_syncobject

Summary of the last successful synchronization from source to destination.

Will always be present if there has been a successful sync. Even if the most recent syncs have failed.

This top-level field is a convenience for consumers who want easy access to last sync information without having to traverse detailed_status.

Show child attributesHide child attributes

sync_start_timestampstring: The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestampstring: The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobjectRequired

Response

Returns the SyncedDatabaseTable object.

View as Markdown

Request

{
  "synced_table": {
    "name": "string",
    "database_instance_name": "string",
    "effective_database_instance_name": "string",
    "logical_database_name": "string",
    "effective_logical_database_name": "string",
    "spec": {},
    "unity_catalog_provisioning_state": "string",
    "data_synchronization_status": {}
  }
}

Response

{
  "name": "string",
  "database_instance_name": "string",
  "effective_database_instance_name": "string",
  "logical_database_name": "string",
  "effective_logical_database_name": "string",
  "spec": {
    "scheduling_policy": "string",
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "existing_pipeline_id": "string",
    "create_database_objects_if_missing": true,
    "new_pipeline_spec": {}
  },
  "unity_catalog_provisioning_state": "string",
  "data_synchronization_status": {
    "detailed_state": "string",
    "message": "string",
    "provisioning_status": {},
    "continuous_update_status": {},
    "triggered_update_status": {},
    "failed_status": {},
    "pipeline_id": "string",
    "last_sync": {}
  }
}

Delete Public Preview

DELETE /api/2.0/database/synced_tables/{name}

Delete a Synced Database Table.

API scopes: postgres

Parameters

namestringpath

purge_databooleanquery: Optional. When set to true, the actual PostgreSQL table will be dropped from the database.