Skip to main content

Synced Table

View as Markdown

A synced table is a read-only Postgres table that Databricks keeps up to date from a Unity Catalog source table through a managed pipeline. It is also registered in Unity Catalog under its three-part name. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables.

SyncedTable object​

A synced table is a read-only Postgres table that Databricks keeps up to date from a Unity Catalog source table through a managed pipeline. It is also registered in Unity Catalog under its three-part name. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables.

namestringIDImmutable

The full resource name of the synced table, where (catalog, schema, table) are the Unity Catalog names of the synced table.

Format: "synced_tables/{catalog}.{schema}.{table}".

For the corresponding source table in Unity Catalog, look for the "source_table_full_name" attribute.

uidstringOutput only

The Unity Catalog table ID for this synced table.

specobjectInput only

Configuration details of the synced table, such as the source table, scheduling policy, etc.

Show child attributesHide child attributes
postgres_databasestring

The name of the Postgres database where the synced table is created. A value must be specified on creation. If the synced table is created inside a Lakebase catalog, the value must match the Postgres database of that catalog.

branchstring

The full resource name of the branch that contains the synced table.

Format: "projects/{project_id}/branches/{branch_id}".

A value must be specified when creating a synced table inside a standard catalog. Inside a Lakebase catalog it can be omitted, and if specified it must match the branch of that catalog.

scheduling_policystring

Scheduling policy of the underlying pipeline. A value must be specified on creation. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables for how to choose a sync mode.

Values:

  • CONTINUOUS
  • TRIGGERED
  • SNAPSHOT
source_table_full_namestring

Three-part (catalog, schema, table) name of the source Delta table.

For the corresponding destination table, use any of the two:

  • synced_table_id used at the creation of the SyncedTable
  • "name" consisting of "synced_tables/" prefix and the full name of the destination table.
primary_key_columnsarray of string

Primary Key columns to be used for data insert/update in the destination. At least one column must be specified.

timeseries_keystring

Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstring

ID of an existing pipeline to bin-pack this synced table into. The pipeline must be a synced table pipeline that runs continuously if and only if scheduling_policy is CONTINUOUS. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned in status.pipeline_id.

create_database_objects_if_missingboolean

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.

Defaults to true if omitted.

Default: true

new_pipeline_specobject

Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned in status.pipeline_id.

Show child attributesHide child attributes
storage_catalogstring

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstring

Budget policy to set on the newly created pipeline.

type_overridesarray of object

Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.

Show child attributesHide child attributes
column_namestring

Name of the source column whose target PostgreSQL type should be overridden.

pg_typestring

PostgreSQL-specific target type to use for the column.

Values:

  • PG_SPECIFIC_TYPE_VECTOR
  • PG_SPECIFIC_TYPE_HALFVEC
  • PG_SPECIFIC_TYPE_VARCHAR
sizeint32

Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.

statusobjectOutput only

Synced Table data synchronization status.

Show child attributesHide child attributes
messagestringOutput only

A text description of the current state of the synced table.

detailed_statestringOutput only

The state of the synced table.

Values:

  • SYNCED_TABLE_PROVISIONING
  • SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES
  • SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT
  • SYNCED_TABLE_ONLINE
  • SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE
  • SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE
  • SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE
  • SYNCED_TABLE_OFFLINE
  • SYNCED_TABLE_OFFLINE_FAILED
  • SYNCED_TABLE_ONLINE_PIPELINE_FAILED
  • SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES
last_syncobjectOutput only

Summary of the last successful synchronization from source to destination.

Show child attributesHide child attributes
sync_start_timestringOutput only

The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestringOutput only

The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobjectOutput only

The source Delta table version that the last successful sync read.

Show child attributesHide child attributes
delta_commit_versionint64Output only

The Delta Lake commit version that was last successfully synced.

delta_commit_timestringOutput only

The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.

ongoing_sync_progressobjectOutput only

Progress of the sync that is currently running, either the initial data load or an update of a triggered or snapshot synced table.

Show child attributesHide child attributes
latest_version_currently_processingint64Output only

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Output only

The number of rows that have been synced in this update.

total_row_countint64Output only

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoubleOutput only

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoubleOutput only

The estimated time remaining to complete this update in seconds.

provisioning_phasestringOutput only

The current phase of the data synchronization pipeline.

Values:

  • PROVISIONING_PHASE_MAIN
  • PROVISIONING_PHASE_INDEX_SCAN
  • PROVISIONING_PHASE_INDEX_SORT
last_processed_commit_versionint64Output only

The last source table Delta version that was successfully synced to the synced table.

last_sync_timestringOutput only

The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

pipeline_idstringOutput only

ID of the associated pipeline.

unity_catalog_provisioning_statestringOutput only

The provisioning state of the synced table entity in Unity Catalog.

Values:

  • PROVISIONING
  • ACTIVE
  • FAILED
  • DELETING
  • UPDATING
  • DEGRADED
projectstringOutput only

The full resource name of the project associated with the table.

Format: "projects/{project_id}".

create_timestringOutput only

The time when the synced table was created.

synced_table_idstringOutput only

The three-part Unity Catalog name of the synced table, in the format "{catalog}.{schema}.{table}". It is the synced_table_id that was used to create the synced table and the last component of name.

Get a Synced Database Table GA​

GET /api/2.0/postgres/{name=synced_tables/*}

Get a Synced Table.

API scopes: postgres

Parameters​

namestringRequiredpath

The Full resource name of the synced table. Format: "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the entity names in the Unity Catalog.

Response​

Returns the SyncedTable object.

Create a Synced Database Table GA​

POST /api/2.0/postgres/synced_tables

Create a Synced Table. Creates the synced table in Unity Catalog and the Postgres table on the branch, and starts the pipeline that loads the initial data from the source table.

API scopes: postgres

Parameters​

synced_table_idstringRequiredquery

The ID to use for the Synced Table. This becomes the final component of the SyncedTable's resource name. ID is required and is the synced table name, containing (catalog, schema, table) tuple. Elements of the tuple are the UC entity names.

Example: "{catalog}.{schema}.{table}"

synced_table_id represents both of the following:

  1. An online VIEW virtual table in the Unity Catalog accessible via the Lakehouse Federation.
  2. Postgres table named "{table}" in schema "{schema}" in the connected Postgres database

Request body​

The synced table to create. Its configuration is read from spec, and its name comes from synced_table_id.

namestringIDImmutable

The full resource name of the synced table, where (catalog, schema, table) are the Unity Catalog names of the synced table.

Format: "synced_tables/{catalog}.{schema}.{table}".

For the corresponding source table in Unity Catalog, look for the "source_table_full_name" attribute.

specobjectInput only

Configuration details of the synced table, such as the source table, scheduling policy, etc.

Show child attributesHide child attributes
postgres_databasestring

The name of the Postgres database where the synced table is created. A value must be specified on creation. If the synced table is created inside a Lakebase catalog, the value must match the Postgres database of that catalog.

branchstring

The full resource name of the branch that contains the synced table.

Format: "projects/{project_id}/branches/{branch_id}".

A value must be specified when creating a synced table inside a standard catalog. Inside a Lakebase catalog it can be omitted, and if specified it must match the branch of that catalog.

scheduling_policystring

Scheduling policy of the underlying pipeline. A value must be specified on creation. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables for how to choose a sync mode.

Values:

  • CONTINUOUS
  • TRIGGERED
  • SNAPSHOT
source_table_full_namestring

Three-part (catalog, schema, table) name of the source Delta table.

For the corresponding destination table, use any of the two:

  • synced_table_id used at the creation of the SyncedTable
  • "name" consisting of "synced_tables/" prefix and the full name of the destination table.
primary_key_columnsarray of string

Primary Key columns to be used for data insert/update in the destination. At least one column must be specified.

timeseries_keystring

Time series key to deduplicate (tie-break) rows with the same primary key.

existing_pipeline_idstring

ID of an existing pipeline to bin-pack this synced table into. The pipeline must be a synced table pipeline that runs continuously if and only if scheduling_policy is CONTINUOUS. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned in status.pipeline_id.

create_database_objects_if_missingboolean

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.

Defaults to true if omitted.

Default: true

new_pipeline_specobject

Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.

The pipeline used for the synced table is returned in status.pipeline_id.

Show child attributesHide child attributes
storage_catalogstring

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstring

Budget policy to set on the newly created pipeline.

type_overridesarray of object

Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.

Show child attributesHide child attributes
column_namestringRequired

Name of the source column whose target PostgreSQL type should be overridden.

pg_typestringRequired

PostgreSQL-specific target type to use for the column.

Values:

  • PG_SPECIFIC_TYPE_VECTOR
  • PG_SPECIFIC_TYPE_HALFVEC
  • PG_SPECIFIC_TYPE_VARCHAR
sizeint32

Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.

Response​

namestring

The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the name should be a resource name ending with operations/{unique_id}.

metadataobject

Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.

doneboolean

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

errorobject

The error result of the operation in case of failure or cancellation.

Show child attributesHide child attributes
error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values
  • UNKNOWN
  • INTERNAL_ERROR
  • TEMPORARILY_UNAVAILABLE
  • IO_ERROR
  • BAD_REQUEST
  • SERVICE_UNDER_MAINTENANCE
  • WORKSPACE_TEMPORARILY_UNAVAILABLE
  • DEADLINE_EXCEEDED
  • CANCELLED
  • RESOURCE_EXHAUSTED
  • ABORTED
  • NOT_FOUND
  • ALREADY_EXISTS
  • UNAUTHENTICATED
  • UNAVAILABLE
  • INVALID_PARAMETER_VALUE
  • ENDPOINT_NOT_FOUND
  • MALFORMED_REQUEST
  • INVALID_STATE
  • PERMISSION_DENIED
  • FEATURE_DISABLED
  • CUSTOMER_UNAUTHORIZED
  • REQUEST_LIMIT_EXCEEDED
  • RESOURCE_CONFLICT
  • UNPARSEABLE_HTTP_ERROR
  • NOT_IMPLEMENTED
  • DATA_LOSS
  • INVALID_STATE_TRANSITION
  • COULD_NOT_ACQUIRE_LOCK
  • RESOURCE_ALREADY_EXISTS
  • RESOURCE_DOES_NOT_EXIST
  • QUOTA_EXCEEDED
  • MAX_BLOCK_SIZE_EXCEEDED
  • MAX_READ_SIZE_EXCEEDED
  • PARTIAL_DELETE
  • MAX_LIST_SIZE_EXCEEDED
  • DRY_RUN_FAILED
  • RESOURCE_LIMIT_EXCEEDED
  • DIRECTORY_NOT_EMPTY
  • DIRECTORY_PROTECTED
  • MAX_NOTEBOOK_SIZE_EXCEEDED
  • MAX_CHILD_NODE_SIZE_EXCEEDED
  • SEARCH_QUERY_TOO_LONG
  • SEARCH_QUERY_TOO_SHORT
  • MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
  • PERMISSION_NOT_PROPAGATED
  • DEPLOYMENT_TIMEOUT
  • GIT_CONFLICT
  • GIT_UNKNOWN_REF
  • GIT_SENSITIVE_TOKEN_DETECTED
  • GIT_URL_NOT_ON_ALLOW_LIST
  • GIT_REMOTE_ERROR
  • PROJECTS_OPERATION_TIMEOUT
  • IPYNB_FILE_IN_REPO
  • INSECURE_PARTNER_RESPONSE
  • MALFORMED_PARTNER_RESPONSE
  • METASTORE_DOES_NOT_EXIST
  • DAC_DOES_NOT_EXIST
  • CATALOG_DOES_NOT_EXIST
  • SCHEMA_DOES_NOT_EXIST
  • TABLE_DOES_NOT_EXIST
  • SHARE_DOES_NOT_EXIST
  • RECIPIENT_DOES_NOT_EXIST
  • STORAGE_CREDENTIAL_DOES_NOT_EXIST
  • EXTERNAL_LOCATION_DOES_NOT_EXIST
  • PRINCIPAL_DOES_NOT_EXIST
  • PROVIDER_DOES_NOT_EXIST
  • METASTORE_ALREADY_EXISTS
  • DAC_ALREADY_EXISTS
  • CATALOG_ALREADY_EXISTS
  • SCHEMA_ALREADY_EXISTS
  • TABLE_ALREADY_EXISTS
  • SHARE_ALREADY_EXISTS
  • RECIPIENT_ALREADY_EXISTS
  • STORAGE_CREDENTIAL_ALREADY_EXISTS
  • EXTERNAL_LOCATION_ALREADY_EXISTS
  • PROVIDER_ALREADY_EXISTS
  • CATALOG_NOT_EMPTY
  • SCHEMA_NOT_EMPTY
  • METASTORE_NOT_EMPTY
  • PROVIDER_SHARE_NOT_ACCESSIBLE
messagestring
stack_tracestring
detailsarray of object
responseobject

The normal, successful response of the operation.

Delete a Synced Database Table GA​

DELETE /api/2.0/postgres/{name=synced_tables/*}

Delete a Synced Table. Drops the Postgres table, removes the synced table from its pipeline, and deletes it from Unity Catalog. The pipeline itself is deleted when no other synced table uses it.

API scopes: postgres

Parameters​

namestringRequiredIDImmutablepath

The Full resource name of the synced table, of the format "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the UC entity names.

Response​

namestring

The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the name should be a resource name ending with operations/{unique_id}.

metadataobject

Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.

doneboolean

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

errorobject

The error result of the operation in case of failure or cancellation.

Show child attributesHide child attributes
error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values
  • UNKNOWN
  • INTERNAL_ERROR
  • TEMPORARILY_UNAVAILABLE
  • IO_ERROR
  • BAD_REQUEST
  • SERVICE_UNDER_MAINTENANCE
  • WORKSPACE_TEMPORARILY_UNAVAILABLE
  • DEADLINE_EXCEEDED
  • CANCELLED
  • RESOURCE_EXHAUSTED
  • ABORTED
  • NOT_FOUND
  • ALREADY_EXISTS
  • UNAUTHENTICATED
  • UNAVAILABLE
  • INVALID_PARAMETER_VALUE
  • ENDPOINT_NOT_FOUND
  • MALFORMED_REQUEST
  • INVALID_STATE
  • PERMISSION_DENIED
  • FEATURE_DISABLED
  • CUSTOMER_UNAUTHORIZED
  • REQUEST_LIMIT_EXCEEDED
  • RESOURCE_CONFLICT
  • UNPARSEABLE_HTTP_ERROR
  • NOT_IMPLEMENTED
  • DATA_LOSS
  • INVALID_STATE_TRANSITION
  • COULD_NOT_ACQUIRE_LOCK
  • RESOURCE_ALREADY_EXISTS
  • RESOURCE_DOES_NOT_EXIST
  • QUOTA_EXCEEDED
  • MAX_BLOCK_SIZE_EXCEEDED
  • MAX_READ_SIZE_EXCEEDED
  • PARTIAL_DELETE
  • MAX_LIST_SIZE_EXCEEDED
  • DRY_RUN_FAILED
  • RESOURCE_LIMIT_EXCEEDED
  • DIRECTORY_NOT_EMPTY
  • DIRECTORY_PROTECTED
  • MAX_NOTEBOOK_SIZE_EXCEEDED
  • MAX_CHILD_NODE_SIZE_EXCEEDED
  • SEARCH_QUERY_TOO_LONG
  • SEARCH_QUERY_TOO_SHORT
  • MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
  • PERMISSION_NOT_PROPAGATED
  • DEPLOYMENT_TIMEOUT
  • GIT_CONFLICT
  • GIT_UNKNOWN_REF
  • GIT_SENSITIVE_TOKEN_DETECTED
  • GIT_URL_NOT_ON_ALLOW_LIST
  • GIT_REMOTE_ERROR
  • PROJECTS_OPERATION_TIMEOUT
  • IPYNB_FILE_IN_REPO
  • INSECURE_PARTNER_RESPONSE
  • MALFORMED_PARTNER_RESPONSE
  • METASTORE_DOES_NOT_EXIST
  • DAC_DOES_NOT_EXIST
  • CATALOG_DOES_NOT_EXIST
  • SCHEMA_DOES_NOT_EXIST
  • TABLE_DOES_NOT_EXIST
  • SHARE_DOES_NOT_EXIST
  • RECIPIENT_DOES_NOT_EXIST
  • STORAGE_CREDENTIAL_DOES_NOT_EXIST
  • EXTERNAL_LOCATION_DOES_NOT_EXIST
  • PRINCIPAL_DOES_NOT_EXIST
  • PROVIDER_DOES_NOT_EXIST
  • METASTORE_ALREADY_EXISTS
  • DAC_ALREADY_EXISTS
  • CATALOG_ALREADY_EXISTS
  • SCHEMA_ALREADY_EXISTS
  • TABLE_ALREADY_EXISTS
  • SHARE_ALREADY_EXISTS
  • RECIPIENT_ALREADY_EXISTS
  • STORAGE_CREDENTIAL_ALREADY_EXISTS
  • EXTERNAL_LOCATION_ALREADY_EXISTS
  • PROVIDER_ALREADY_EXISTS
  • CATALOG_NOT_EMPTY
  • SCHEMA_NOT_EMPTY
  • METASTORE_NOT_EMPTY
  • PROVIDER_SHARE_NOT_ACCESSIBLE
messagestring
stack_tracestring
detailsarray of object
responseobject

The normal, successful response of the operation.