Synced Table

View as Markdown

GetGET /api/2.0/postgres/{name=synced_tables/*}CreatePOST /api/2.0/postgres/synced_tablesDeleteDELETE /api/2.0/postgres/{name=synced_tables/*}

SyncedTable object

namestring

Output only. The Full resource name of the synced table in Postgres where (catalog, schema, table) are the UC entity names.

Format "synced_tables/{catalog}.{schema}.{table}"

For the corresponding source table in the Unity catalog look for the "source_table_full_name" attribute.

uidstring

The Unity Catalog table ID for this synced table.

specobject

Configuration details of the synced table, such as the source table, scheduling policy, etc. This attribute is specified at creation time and most fields are returned as is on subsequent queries.

Show child attributesHide child attributes

postgres_databasestring

The Postgres database name where the synced table will be created in.

If this synced table is created inside a Lakebase Catalog, this attribute can be omitted on creation and is inferred from the postgres_database associated with the Lakebase Catalog. If specified when inside a Lakebase Catalog, the value must match.

A value must be specified when creating a synced table inside a Standard Catalog.

branchstring

The full resource name the branch associated with the table.

Format: "projects/{project_id}/branches/{branch_id}".

scheduling_policystring

Scheduling policy of the underlying pipeline.

Values: SYNCED_TABLE_SCHEDULING_POLICY_UNSPECIFIED, CONTINUOUS, TRIGGERED, SNAPSHOT

source_table_full_namestring

Three-part (catalog, schema, table) name of the source Delta table.

For the corresponding destination table, use any of the two:

• synced_table_id used at the creation of the SyncedTable
• "name" consisting of "synced_tables/" prefix and the full name of the destination table.

primary_key_columnsarray of string

Primary Key columns to be used for data insert/update in the destination.

timeseries_keystring

Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstring

ID of an existing pipeline to bin-pack this synced table into. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned via the top level pipeline_id attribute.

create_database_objects_if_missingboolean

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.

Defaults to true if omitted.

Default: true

new_pipeline_specobject

Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned via the top level pipeline_id attribute.

Show child attributesHide child attributes

storage_catalogstring

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstring

Budget policy to set on the newly created pipeline.

type_overridesarray of object

Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.

Show child attributesHide child attributes

column_namestring

Name of the source column whose target PostgreSQL type should be overridden.

pg_typestring

PostgreSQL-specific target type to use for the column.

Values: PG_SPECIFIC_TYPE_UNSPECIFIED, PG_SPECIFIC_TYPE_VECTOR, PG_SPECIFIC_TYPE_HALFVEC, PG_SPECIFIC_TYPE_VARCHAR

sizeint32

Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.

statusobject

Synced Table data synchronization status.

Show child attributesHide child attributes

messagestring

A text description of the current state of the synced table.

detailed_statestring

The state of the synced table.

Values: SYNCED_TABLE_STATE_UNSPECIFIED, SYNCED_TABLE_PROVISIONING, SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES, SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT, SYNCED_TABLE_ONLINE, SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE, SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE, SYNCED_TABLE_OFFLINE, SYNCED_TABLE_OFFLINE_FAILED, SYNCED_TABLE_ONLINE_PIPELINE_FAILED, SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES

last_syncobject

Summary of the last successful synchronization from source to destination.

Show child attributesHide child attributes

sync_start_timestring

The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestring

The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobject

Show child attributesHide child attributes

delta_commit_versionint64

The Delta Lake commit version that was last successfully synced.

delta_commit_timestring

The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.

ongoing_sync_progressobject

Show child attributesHide child attributes

latest_version_currently_processingint64

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64

The number of rows that have been synced in this update.

total_row_countint64

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondouble

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdouble

The estimated time remaining to complete this update in seconds.

provisioning_phasestring

The current phase of the data synchronization pipeline.

Values: PROVISIONING_PHASE_UNSPECIFIED, PROVISIONING_PHASE_MAIN, PROVISIONING_PHASE_INDEX_SCAN, PROVISIONING_PHASE_INDEX_SORT

last_processed_commit_versionint64

The last source table Delta version that was successfully synced to the synced table.

last_sync_timestring

The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

pipeline_idstring

ID of the associated pipeline.

unity_catalog_provisioning_statestring

The provisioning state of the synced table entity in Unity Catalog.

Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

projectstring

The full resource name of the project associated with the table.

Format: "projects/{project_id}".

create_timestring

synced_table_idstring

The part of the name, chosen by the user when the resource was created.

View as Markdown

The SyncedTable object

{
  "name": "string",
  "uid": "string",
  "spec": {
    "postgres_database": "string",
    "branch": "string",
    "scheduling_policy": "string",
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "existing_pipeline_id": "string",
    "create_database_objects_if_missing": true,
    "new_pipeline_spec": {},
    "type_overrides": [
      {}
    ]
  },
  "status": {
    "message": "string",
    "detailed_state": "string",
    "last_sync": {},
    "ongoing_sync_progress": {},
    "provisioning_phase": "string",
    "last_processed_commit_version": 0,
    "last_sync_time": "string",
    "pipeline_id": "string",
    "unity_catalog_provisioning_state": "string",
    "project": "string"
  },
  "create_time": "string",
  "synced_table_id": "string"
}

Get GA

GET /api/2.0/postgres/{name=synced_tables/*}

Get a Synced Table.

API scopes: postgres

Parameters

namestringpath

The Full resource name of the synced table. Format: "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the entity names in the Unity Catalog.

Response

Returns the SyncedTable object.

View as Markdown

Response

{
  "name": "string",
  "uid": "string",
  "spec": {
    "postgres_database": "string",
    "branch": "string",
    "scheduling_policy": "string",
    "source_table_full_name": "string",
    "primary_key_columns": [
      "string"
    ],
    "timeseries_key": "string",
    "existing_pipeline_id": "string",
    "create_database_objects_if_missing": true,
    "new_pipeline_spec": {},
    "type_overrides": [
      {}
    ]
  },
  "status": {
    "message": "string",
    "detailed_state": "string",
    "last_sync": {},
    "ongoing_sync_progress": {},
    "provisioning_phase": "string",
    "last_processed_commit_version": 0,
    "last_sync_time": "string",
    "pipeline_id": "string",
    "unity_catalog_provisioning_state": "string",
    "project": "string"
  },
  "create_time": "string",
  "synced_table_id": "string"
}

Create GA

POST /api/2.0/postgres/synced_tables

Create a Synced Table.

API scopes: postgres

Parameters

synced_table_idstringquery

The ID to use for the Synced Table. This becomes the final component of the SyncedTable's resource name. ID is required and is the synced table name, containing (catalog, schema, table) tuple. Elements of the tuple are the UC entity names.

Example: "{catalog}.{schema}.{table}"

synced_table_id represents both of the following:

1. An online VIEW virtual table in the Unity Catalog accessible via the Lakehouse Federation.
2. Postgres table named "{table}" in schema "{schema}" in the connected Postgres database

Request body

synced_tableobject

Show child attributesHide child attributes

namestring

Output only. The Full resource name of the synced table in Postgres where (catalog, schema, table) are the UC entity names.

Format "synced_tables/{catalog}.{schema}.{table}"

For the corresponding source table in the Unity catalog look for the "source_table_full_name" attribute.

uidstring

The Unity Catalog table ID for this synced table.

specobject

Configuration details of the synced table, such as the source table, scheduling policy, etc. This attribute is specified at creation time and most fields are returned as is on subsequent queries.

Show child attributesHide child attributes

postgres_databasestring

The Postgres database name where the synced table will be created in.

If this synced table is created inside a Lakebase Catalog, this attribute can be omitted on creation and is inferred from the postgres_database associated with the Lakebase Catalog. If specified when inside a Lakebase Catalog, the value must match.

A value must be specified when creating a synced table inside a Standard Catalog.

branchstring

The full resource name the branch associated with the table.

Format: "projects/{project_id}/branches/{branch_id}".

scheduling_policystring

Scheduling policy of the underlying pipeline.

Values: SYNCED_TABLE_SCHEDULING_POLICY_UNSPECIFIED, CONTINUOUS, TRIGGERED, SNAPSHOT

source_table_full_namestring

Three-part (catalog, schema, table) name of the source Delta table.

For the corresponding destination table, use any of the two:

• synced_table_id used at the creation of the SyncedTable
• "name" consisting of "synced_tables/" prefix and the full name of the destination table.

primary_key_columnsarray of string

Primary Key columns to be used for data insert/update in the destination.

timeseries_keystring

Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstring

ID of an existing pipeline to bin-pack this synced table into. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned via the top level pipeline_id attribute.

create_database_objects_if_missingboolean

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.

Defaults to true if omitted.

Default: true

new_pipeline_specobject

Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned via the top level pipeline_id attribute.

Show child attributesHide child attributes

storage_catalogstring

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstring

Budget policy to set on the newly created pipeline.

type_overridesarray of object

Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.

Show child attributesHide child attributes

column_namestring

Name of the source column whose target PostgreSQL type should be overridden.

pg_typestring

PostgreSQL-specific target type to use for the column.

Values: PG_SPECIFIC_TYPE_UNSPECIFIED, PG_SPECIFIC_TYPE_VECTOR, PG_SPECIFIC_TYPE_HALFVEC, PG_SPECIFIC_TYPE_VARCHAR

sizeint32

Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.

statusobject

Synced Table data synchronization status.

Show child attributesHide child attributes

messagestring

A text description of the current state of the synced table.

detailed_statestring

The state of the synced table.

Values: SYNCED_TABLE_STATE_UNSPECIFIED, SYNCED_TABLE_PROVISIONING, SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES, SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT, SYNCED_TABLE_ONLINE, SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE, SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE, SYNCED_TABLE_OFFLINE, SYNCED_TABLE_OFFLINE_FAILED, SYNCED_TABLE_ONLINE_PIPELINE_FAILED, SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES

last_syncobject

Summary of the last successful synchronization from source to destination.

Show child attributesHide child attributes

sync_start_timestring

The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestring

The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobjectRequired

ongoing_sync_progressobject

Show child attributesHide child attributes

latest_version_currently_processingint64

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64

The number of rows that have been synced in this update.

total_row_countint64

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondouble

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdouble

The estimated time remaining to complete this update in seconds.

provisioning_phasestring

The current phase of the data synchronization pipeline.

Values: PROVISIONING_PHASE_UNSPECIFIED, PROVISIONING_PHASE_MAIN, PROVISIONING_PHASE_INDEX_SCAN, PROVISIONING_PHASE_INDEX_SORT

last_processed_commit_versionint64

The last source table Delta version that was successfully synced to the synced table.

last_sync_timestring

The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

pipeline_idstring

ID of the associated pipeline.

unity_catalog_provisioning_statestring

The provisioning state of the synced table entity in Unity Catalog.

Values: STATE_UNSPECIFIED, PROVISIONING, ACTIVE, FAILED, DELETING, UPDATING, DEGRADED

projectstring

The full resource name of the project associated with the table.

Format: "projects/{project_id}".

create_timestring

synced_table_idstring

The part of the name, chosen by the user when the resource was created.

Response

namestring

The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the name should be a resource name ending with operations/{unique_id}.

metadataobject

Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.

doneboolean

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

errorobjectRequired

The error result of the operation in case of failure or cancellation.

Show child attributesHide child attributes

error_codestring

Values: UNKNOWN, INTERNAL_ERROR, TEMPORARILY_UNAVAILABLE, IO_ERROR, BAD_REQUEST, SERVICE_UNDER_MAINTENANCE, WORKSPACE_TEMPORARILY_UNAVAILABLE, DEADLINE_EXCEEDED, CANCELLED, RESOURCE_EXHAUSTED, ABORTED, NOT_FOUND, ALREADY_EXISTS, UNAUTHENTICATED, UNAVAILABLE, INVALID_PARAMETER_VALUE, ENDPOINT_NOT_FOUND, MALFORMED_REQUEST, INVALID_STATE, PERMISSION_DENIED, FEATURE_DISABLED, CUSTOMER_UNAUTHORIZED, REQUEST_LIMIT_EXCEEDED, RESOURCE_CONFLICT, UNPARSEABLE_HTTP_ERROR, NOT_IMPLEMENTED, DATA_LOSS, INVALID_STATE_TRANSITION, COULD_NOT_ACQUIRE_LOCK, RESOURCE_ALREADY_EXISTS, RESOURCE_DOES_NOT_EXIST, QUOTA_EXCEEDED, MAX_BLOCK_SIZE_EXCEEDED, MAX_READ_SIZE_EXCEEDED, PARTIAL_DELETE, MAX_LIST_SIZE_EXCEEDED, DRY_RUN_FAILED, RESOURCE_LIMIT_EXCEEDED, DIRECTORY_NOT_EMPTY, DIRECTORY_PROTECTED, MAX_NOTEBOOK_SIZE_EXCEEDED, MAX_CHILD_NODE_SIZE_EXCEEDED, SEARCH_QUERY_TOO_LONG, SEARCH_QUERY_TOO_SHORT, MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST, PERMISSION_NOT_PROPAGATED, DEPLOYMENT_TIMEOUT, GIT_CONFLICT, GIT_UNKNOWN_REF, GIT_SENSITIVE_TOKEN_DETECTED, GIT_URL_NOT_ON_ALLOW_LIST, GIT_REMOTE_ERROR, PROJECTS_OPERATION_TIMEOUT, IPYNB_FILE_IN_REPO, INSECURE_PARTNER_RESPONSE, MALFORMED_PARTNER_RESPONSE, METASTORE_DOES_NOT_EXIST, DAC_DOES_NOT_EXIST, CATALOG_DOES_NOT_EXIST, SCHEMA_DOES_NOT_EXIST, TABLE_DOES_NOT_EXIST, SHARE_DOES_NOT_EXIST, RECIPIENT_DOES_NOT_EXIST, STORAGE_CREDENTIAL_DOES_NOT_EXIST, EXTERNAL_LOCATION_DOES_NOT_EXIST, PRINCIPAL_DOES_NOT_EXIST, PROVIDER_DOES_NOT_EXIST, METASTORE_ALREADY_EXISTS, DAC_ALREADY_EXISTS, CATALOG_ALREADY_EXISTS, SCHEMA_ALREADY_EXISTS, TABLE_ALREADY_EXISTS, SHARE_ALREADY_EXISTS, RECIPIENT_ALREADY_EXISTS, STORAGE_CREDENTIAL_ALREADY_EXISTS, EXTERNAL_LOCATION_ALREADY_EXISTS, PROVIDER_ALREADY_EXISTS, CATALOG_NOT_EMPTY, SCHEMA_NOT_EMPTY, METASTORE_NOT_EMPTY, PROVIDER_SHARE_NOT_ACCESSIBLE

messagestring

stack_tracestring

detailsarray of object

responseobjectRequired

The normal, successful response of the operation.

View as Markdown

Request

{
  "synced_table": {
    "name": "string",
    "uid": "string",
    "spec": {},
    "status": {},
    "create_time": "string",
    "synced_table_id": "string"
  }
}

Response

{
  "name": "string",
  "metadata": {},
  "done": true,
  "error": {
    "error_code": "string",
    "message": "string",
    "stack_trace": "string",
    "details": [
      {}
    ]
  },
  "response": {}
}

Delete GA

DELETE /api/2.0/postgres/{name=synced_tables/*}

Delete a Synced Table.

API scopes: postgres

Parameters

namestringpath

The Full resource name of the synced table, of the format "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the UC entity names.

Response

namestring

The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the name should be a resource name ending with operations/{unique_id}.

metadataobject

Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.

doneboolean

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

errorobjectRequired

The error result of the operation in case of failure or cancellation.

Show child attributesHide child attributes

error_codestring

Values: UNKNOWN, INTERNAL_ERROR, TEMPORARILY_UNAVAILABLE, IO_ERROR, BAD_REQUEST, SERVICE_UNDER_MAINTENANCE, WORKSPACE_TEMPORARILY_UNAVAILABLE, DEADLINE_EXCEEDED, CANCELLED, RESOURCE_EXHAUSTED, ABORTED, NOT_FOUND, ALREADY_EXISTS, UNAUTHENTICATED, UNAVAILABLE, INVALID_PARAMETER_VALUE, ENDPOINT_NOT_FOUND, MALFORMED_REQUEST, INVALID_STATE, PERMISSION_DENIED, FEATURE_DISABLED, CUSTOMER_UNAUTHORIZED, REQUEST_LIMIT_EXCEEDED, RESOURCE_CONFLICT, UNPARSEABLE_HTTP_ERROR, NOT_IMPLEMENTED, DATA_LOSS, INVALID_STATE_TRANSITION, COULD_NOT_ACQUIRE_LOCK, RESOURCE_ALREADY_EXISTS, RESOURCE_DOES_NOT_EXIST, QUOTA_EXCEEDED, MAX_BLOCK_SIZE_EXCEEDED, MAX_READ_SIZE_EXCEEDED, PARTIAL_DELETE, MAX_LIST_SIZE_EXCEEDED, DRY_RUN_FAILED, RESOURCE_LIMIT_EXCEEDED, DIRECTORY_NOT_EMPTY, DIRECTORY_PROTECTED, MAX_NOTEBOOK_SIZE_EXCEEDED, MAX_CHILD_NODE_SIZE_EXCEEDED, SEARCH_QUERY_TOO_LONG, SEARCH_QUERY_TOO_SHORT, MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST, PERMISSION_NOT_PROPAGATED, DEPLOYMENT_TIMEOUT, GIT_CONFLICT, GIT_UNKNOWN_REF, GIT_SENSITIVE_TOKEN_DETECTED, GIT_URL_NOT_ON_ALLOW_LIST, GIT_REMOTE_ERROR, PROJECTS_OPERATION_TIMEOUT, IPYNB_FILE_IN_REPO, INSECURE_PARTNER_RESPONSE, MALFORMED_PARTNER_RESPONSE, METASTORE_DOES_NOT_EXIST, DAC_DOES_NOT_EXIST, CATALOG_DOES_NOT_EXIST, SCHEMA_DOES_NOT_EXIST, TABLE_DOES_NOT_EXIST, SHARE_DOES_NOT_EXIST, RECIPIENT_DOES_NOT_EXIST, STORAGE_CREDENTIAL_DOES_NOT_EXIST, EXTERNAL_LOCATION_DOES_NOT_EXIST, PRINCIPAL_DOES_NOT_EXIST, PROVIDER_DOES_NOT_EXIST, METASTORE_ALREADY_EXISTS, DAC_ALREADY_EXISTS, CATALOG_ALREADY_EXISTS, SCHEMA_ALREADY_EXISTS, TABLE_ALREADY_EXISTS, SHARE_ALREADY_EXISTS, RECIPIENT_ALREADY_EXISTS, STORAGE_CREDENTIAL_ALREADY_EXISTS, EXTERNAL_LOCATION_ALREADY_EXISTS, PROVIDER_ALREADY_EXISTS, CATALOG_NOT_EMPTY, SCHEMA_NOT_EMPTY, METASTORE_NOT_EMPTY, PROVIDER_SHARE_NOT_ACCESSIBLE

messagestring

stack_tracestring

detailsarray of object

responseobjectRequired

The normal, successful response of the operation.

View as Markdown

Response

{
  "name": "string",
  "metadata": {},
  "done": true,
  "error": {
    "error_code": "string",
    "message": "string",
    "stack_trace": "string",
    "details": [
      {}
    ]
  },
  "response": {}
}