Synced Table
A synced table is a read-only Postgres table that Databricks keeps up to date from a Unity Catalog source table through a managed pipeline. It is also registered in Unity Catalog under its three-part name. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables.
SyncedTable object
A synced table is a read-only Postgres table that Databricks keeps up to date from a Unity Catalog source table through a managed pipeline. It is also registered in Unity Catalog under its three-part name. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables.
- namestringIDImmutable
The full resource name of the synced table, where (catalog, schema, table) are the Unity Catalog names of the synced table.
Format: "synced_tables/{catalog}.{schema}.{table}".
For the corresponding source table in Unity Catalog, look for the "source_table_full_name" attribute.
- uidstringOutput only
The Unity Catalog table ID for this synced table.
- specobjectInput only
Configuration details of the synced table, such as the source table, scheduling policy, etc.
Show child attributesHide child attributes
- postgres_databasestring
The name of the Postgres database where the synced table is created. A value must be specified on creation. If the synced table is created inside a Lakebase catalog, the value must match the Postgres database of that catalog.
- branchstring
The full resource name of the branch that contains the synced table.
Format: "projects/{project_id}/branches/{branch_id}".
A value must be specified when creating a synced table inside a standard catalog. Inside a Lakebase catalog it can be omitted, and if specified it must match the branch of that catalog.
- scheduling_policystring
Scheduling policy of the underlying pipeline. A value must be specified on creation. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables for how to choose a sync mode.
CONTINUOUSTRIGGEREDSNAPSHOT
- source_table_full_namestring
Three-part (catalog, schema, table) name of the source Delta table.
For the corresponding destination table, use any of the two:
- synced_table_id used at the creation of the SyncedTable
- "name" consisting of "synced_tables/" prefix and the full name of the destination table.
- primary_key_columnsarray of string
Primary Key columns to be used for data insert/update in the destination. At least one column must be specified.
- timeseries_keystring
Time series key to deduplicate (tie-break) rows with the same primary key.
- existing_pipeline_idstring
ID of an existing pipeline to bin-pack this synced table into. The pipeline must be a synced table pipeline that runs continuously if and only if scheduling_policy is CONTINUOUS. At most one of existing_pipeline_id and new_pipeline_spec should be defined.
The pipeline used for the synced table is returned in status.pipeline_id.
- create_database_objects_if_missingboolean
If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.
Defaults to true if omitted.
- new_pipeline_specobject
Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.
The pipeline used for the synced table is returned in status.pipeline_id.
Show child attributesHide child attributes
- storage_catalogstring
UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.
- storage_schemastring
UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.
- budget_policy_idstring
Budget policy to set on the newly created pipeline.
- type_overridesarray of object
Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.
Show child attributesHide child attributes
- column_namestring
Name of the source column whose target PostgreSQL type should be overridden.
- pg_typestring
PostgreSQL-specific target type to use for the column.
PG_SPECIFIC_TYPE_VECTORPG_SPECIFIC_TYPE_HALFVECPG_SPECIFIC_TYPE_VARCHAR
- sizeint32
Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.
- statusobjectOutput only
Synced Table data synchronization status.
Show child attributesHide child attributes
- messagestringOutput only
A text description of the current state of the synced table.
- detailed_statestringOutput only
The state of the synced table.
SYNCED_TABLE_PROVISIONINGSYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCESSYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOTSYNCED_TABLE_ONLINESYNCED_TABLE_ONLINE_CONTINUOUS_UPDATESYNCED_TABLE_ONLINE_TRIGGERED_UPDATESYNCED_TABLE_ONLINE_NO_PENDING_UPDATESYNCED_TABLE_OFFLINESYNCED_TABLE_OFFLINE_FAILEDSYNCED_TABLE_ONLINE_PIPELINE_FAILEDSYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES
- last_syncobjectOutput only
Summary of the last successful synchronization from source to destination.
Show child attributesHide child attributes
- sync_start_timestringOutput only
The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.
- sync_end_timestringOutput only
The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.
- delta_table_sync_infoobjectOutput only
The source Delta table version that the last successful sync read.
Show child attributesHide child attributes
- delta_commit_versionint64Output only
The Delta Lake commit version that was last successfully synced.
- delta_commit_timestringOutput only
The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.
- ongoing_sync_progressobjectOutput only
Progress of the sync that is currently running, either the initial data load or an update of a triggered or snapshot synced table.
Show child attributesHide child attributes
- latest_version_currently_processingint64Output only
The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.
- synced_row_countint64Output only
The number of rows that have been synced in this update.
- total_row_countint64Output only
The total number of rows that need to be synced in this update. This number may be an estimate.
- sync_progress_completiondoubleOutput only
The completion ratio of this update. This is a number between 0 and 1.
- estimated_completion_time_secondsdoubleOutput only
The estimated time remaining to complete this update in seconds.
- provisioning_phasestringOutput only
The current phase of the data synchronization pipeline.
PROVISIONING_PHASE_MAINPROVISIONING_PHASE_INDEX_SCANPROVISIONING_PHASE_INDEX_SORT
- last_processed_commit_versionint64Output only
The last source table Delta version that was successfully synced to the synced table.
- last_sync_timestringOutput only
The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.
- pipeline_idstringOutput only
ID of the associated pipeline.
- unity_catalog_provisioning_statestringOutput only
The provisioning state of the synced table entity in Unity Catalog.
PROVISIONINGACTIVEFAILEDDELETINGUPDATINGDEGRADED
- projectstringOutput only
The full resource name of the project associated with the table.
Format: "projects/{project_id}".
- create_timestringOutput only
The time when the synced table was created.
- synced_table_idstringOutput only
The three-part Unity Catalog name of the synced table, in the format "{catalog}.{schema}.{table}". It is the synced_table_id that was used to create the synced table and the last component of name.
Get a Synced Database Table GA
GET
Get a Synced Table.
API scopes: postgres
Parameters
- namestringRequiredpath
The Full resource name of the synced table. Format: "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the entity names in the Unity Catalog.
Response
Returns the SyncedTable object.
Create a Synced Database Table GA
POST
Create a Synced Table. Creates the synced table in Unity Catalog and the Postgres table on the branch, and starts the pipeline that loads the initial data from the source table.
API scopes: postgres
Parameters
- synced_table_idstringRequiredquery
The ID to use for the Synced Table. This becomes the final component of the SyncedTable's resource name. ID is required and is the synced table name, containing (catalog, schema, table) tuple. Elements of the tuple are the UC entity names.
Example: "{catalog}.{schema}.{table}"
synced_table_id represents both of the following:
- An online VIEW virtual table in the Unity Catalog accessible via the Lakehouse Federation.
- Postgres table named "{table}" in schema "{schema}" in the connected Postgres database
Request body
The synced table to create. Its configuration is read from spec, and its name comes from synced_table_id.
- namestringIDImmutable
The full resource name of the synced table, where (catalog, schema, table) are the Unity Catalog names of the synced table.
Format: "synced_tables/{catalog}.{schema}.{table}".
For the corresponding source table in Unity Catalog, look for the "source_table_full_name" attribute.
- specobjectInput only
Configuration details of the synced table, such as the source table, scheduling policy, etc.
Show child attributesHide child attributes
- postgres_databasestring
The name of the Postgres database where the synced table is created. A value must be specified on creation. If the synced table is created inside a Lakebase catalog, the value must match the Postgres database of that catalog.
- branchstring
The full resource name of the branch that contains the synced table.
Format: "projects/{project_id}/branches/{branch_id}".
A value must be specified when creating a synced table inside a standard catalog. Inside a Lakebase catalog it can be omitted, and if specified it must match the branch of that catalog.
- scheduling_policystring
Scheduling policy of the underlying pipeline. A value must be specified on creation. See https://docs.databricks.com/aws/en/oltp/projects/sync-tables for how to choose a sync mode.
CONTINUOUSTRIGGEREDSNAPSHOT
- source_table_full_namestring
Three-part (catalog, schema, table) name of the source Delta table.
For the corresponding destination table, use any of the two:
- synced_table_id used at the creation of the SyncedTable
- "name" consisting of "synced_tables/" prefix and the full name of the destination table.
- primary_key_columnsarray of string
Primary Key columns to be used for data insert/update in the destination. At least one column must be specified.
- timeseries_keystring
Time series key to deduplicate (tie-break) rows with the same primary key.
- existing_pipeline_idstring
ID of an existing pipeline to bin-pack this synced table into. The pipeline must be a synced table pipeline that runs continuously if and only if scheduling_policy is CONTINUOUS. At most one of existing_pipeline_id and new_pipeline_spec should be defined.
The pipeline used for the synced table is returned in status.pipeline_id.
- create_database_objects_if_missingboolean
If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request will fail if this is false and the database/schema do not exist.
Defaults to true if omitted.
- new_pipeline_specobject
Specification for creating a new pipeline. At most one of existing_pipeline_id and new_pipeline_spec should be defined.
The pipeline used for the synced table is returned in status.pipeline_id.
Show child attributesHide child attributes
- storage_catalogstring
UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.
- storage_schemastring
UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.
- budget_policy_idstring
Budget policy to set on the newly created pipeline.
- type_overridesarray of object
Override the default Delta->PG type mapping for specific columns. A TypeOverride with PG_SPECIFIC_TYPE_UNSPECIFIED is rejected; a valid pg_type must be set.
Show child attributesHide child attributes
- column_namestringRequired
Name of the source column whose target PostgreSQL type should be overridden.
- pg_typestringRequired
PostgreSQL-specific target type to use for the column.
PG_SPECIFIC_TYPE_VECTORPG_SPECIFIC_TYPE_HALFVECPG_SPECIFIC_TYPE_VARCHAR
- sizeint32
Size parameter for the target type, for types that take one (e.g. vector dimension, varchar length). Required when the chosen pg_type needs a size.
Response
- namestring
The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the
nameshould be a resource name ending withoperations/{unique_id}.
- metadataobject
Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.
- doneboolean
If the value is
false, it means the operation is still in progress. Iftrue, the operation is completed, and eithererrororresponseis available.
- errorobject
The error result of the operation in case of failure or cancellation.
Show child attributesHide child attributes
- error_codestring
Error codes returned by Databricks APIs to indicate specific failure conditions.
Show 81 possible values
UNKNOWNINTERNAL_ERRORTEMPORARILY_UNAVAILABLEIO_ERRORBAD_REQUESTSERVICE_UNDER_MAINTENANCEWORKSPACE_TEMPORARILY_UNAVAILABLEDEADLINE_EXCEEDEDCANCELLEDRESOURCE_EXHAUSTEDABORTEDNOT_FOUNDALREADY_EXISTSUNAUTHENTICATEDUNAVAILABLEINVALID_PARAMETER_VALUEENDPOINT_NOT_FOUNDMALFORMED_REQUESTINVALID_STATEPERMISSION_DENIEDFEATURE_DISABLEDCUSTOMER_UNAUTHORIZEDREQUEST_LIMIT_EXCEEDEDRESOURCE_CONFLICTUNPARSEABLE_HTTP_ERRORNOT_IMPLEMENTEDDATA_LOSSINVALID_STATE_TRANSITIONCOULD_NOT_ACQUIRE_LOCKRESOURCE_ALREADY_EXISTSRESOURCE_DOES_NOT_EXISTQUOTA_EXCEEDEDMAX_BLOCK_SIZE_EXCEEDEDMAX_READ_SIZE_EXCEEDEDPARTIAL_DELETEMAX_LIST_SIZE_EXCEEDEDDRY_RUN_FAILEDRESOURCE_LIMIT_EXCEEDEDDIRECTORY_NOT_EMPTYDIRECTORY_PROTECTEDMAX_NOTEBOOK_SIZE_EXCEEDEDMAX_CHILD_NODE_SIZE_EXCEEDEDSEARCH_QUERY_TOO_LONGSEARCH_QUERY_TOO_SHORTMANAGED_RESOURCE_GROUP_DOES_NOT_EXISTPERMISSION_NOT_PROPAGATEDDEPLOYMENT_TIMEOUTGIT_CONFLICTGIT_UNKNOWN_REFGIT_SENSITIVE_TOKEN_DETECTEDGIT_URL_NOT_ON_ALLOW_LISTGIT_REMOTE_ERRORPROJECTS_OPERATION_TIMEOUTIPYNB_FILE_IN_REPOINSECURE_PARTNER_RESPONSEMALFORMED_PARTNER_RESPONSEMETASTORE_DOES_NOT_EXISTDAC_DOES_NOT_EXISTCATALOG_DOES_NOT_EXISTSCHEMA_DOES_NOT_EXISTTABLE_DOES_NOT_EXISTSHARE_DOES_NOT_EXISTRECIPIENT_DOES_NOT_EXISTSTORAGE_CREDENTIAL_DOES_NOT_EXISTEXTERNAL_LOCATION_DOES_NOT_EXISTPRINCIPAL_DOES_NOT_EXISTPROVIDER_DOES_NOT_EXISTMETASTORE_ALREADY_EXISTSDAC_ALREADY_EXISTSCATALOG_ALREADY_EXISTSSCHEMA_ALREADY_EXISTSTABLE_ALREADY_EXISTSSHARE_ALREADY_EXISTSRECIPIENT_ALREADY_EXISTSSTORAGE_CREDENTIAL_ALREADY_EXISTSEXTERNAL_LOCATION_ALREADY_EXISTSPROVIDER_ALREADY_EXISTSCATALOG_NOT_EMPTYSCHEMA_NOT_EMPTYMETASTORE_NOT_EMPTYPROVIDER_SHARE_NOT_ACCESSIBLE
- messagestring
- stack_tracestring
- detailsarray of object
- responseobject
The normal, successful response of the operation.
Delete a Synced Database Table GA
DELETE
Delete a Synced Table. Drops the Postgres table, removes the synced table from its pipeline, and deletes it from Unity Catalog. The pipeline itself is deleted when no other synced table uses it.
API scopes: postgres
Parameters
- namestringRequiredIDImmutablepath
The Full resource name of the synced table, of the format "synced_tables/{catalog}.{schema}.{table}", where (catalog, schema, table) are the UC entity names.
Response
- namestring
The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the
nameshould be a resource name ending withoperations/{unique_id}.
- metadataobject
Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata.
- doneboolean
If the value is
false, it means the operation is still in progress. Iftrue, the operation is completed, and eithererrororresponseis available.
- errorobject
The error result of the operation in case of failure or cancellation.
Show child attributesHide child attributes
- error_codestring
Error codes returned by Databricks APIs to indicate specific failure conditions.
Show 81 possible values
UNKNOWNINTERNAL_ERRORTEMPORARILY_UNAVAILABLEIO_ERRORBAD_REQUESTSERVICE_UNDER_MAINTENANCEWORKSPACE_TEMPORARILY_UNAVAILABLEDEADLINE_EXCEEDEDCANCELLEDRESOURCE_EXHAUSTEDABORTEDNOT_FOUNDALREADY_EXISTSUNAUTHENTICATEDUNAVAILABLEINVALID_PARAMETER_VALUEENDPOINT_NOT_FOUNDMALFORMED_REQUESTINVALID_STATEPERMISSION_DENIEDFEATURE_DISABLEDCUSTOMER_UNAUTHORIZEDREQUEST_LIMIT_EXCEEDEDRESOURCE_CONFLICTUNPARSEABLE_HTTP_ERRORNOT_IMPLEMENTEDDATA_LOSSINVALID_STATE_TRANSITIONCOULD_NOT_ACQUIRE_LOCKRESOURCE_ALREADY_EXISTSRESOURCE_DOES_NOT_EXISTQUOTA_EXCEEDEDMAX_BLOCK_SIZE_EXCEEDEDMAX_READ_SIZE_EXCEEDEDPARTIAL_DELETEMAX_LIST_SIZE_EXCEEDEDDRY_RUN_FAILEDRESOURCE_LIMIT_EXCEEDEDDIRECTORY_NOT_EMPTYDIRECTORY_PROTECTEDMAX_NOTEBOOK_SIZE_EXCEEDEDMAX_CHILD_NODE_SIZE_EXCEEDEDSEARCH_QUERY_TOO_LONGSEARCH_QUERY_TOO_SHORTMANAGED_RESOURCE_GROUP_DOES_NOT_EXISTPERMISSION_NOT_PROPAGATEDDEPLOYMENT_TIMEOUTGIT_CONFLICTGIT_UNKNOWN_REFGIT_SENSITIVE_TOKEN_DETECTEDGIT_URL_NOT_ON_ALLOW_LISTGIT_REMOTE_ERRORPROJECTS_OPERATION_TIMEOUTIPYNB_FILE_IN_REPOINSECURE_PARTNER_RESPONSEMALFORMED_PARTNER_RESPONSEMETASTORE_DOES_NOT_EXISTDAC_DOES_NOT_EXISTCATALOG_DOES_NOT_EXISTSCHEMA_DOES_NOT_EXISTTABLE_DOES_NOT_EXISTSHARE_DOES_NOT_EXISTRECIPIENT_DOES_NOT_EXISTSTORAGE_CREDENTIAL_DOES_NOT_EXISTEXTERNAL_LOCATION_DOES_NOT_EXISTPRINCIPAL_DOES_NOT_EXISTPROVIDER_DOES_NOT_EXISTMETASTORE_ALREADY_EXISTSDAC_ALREADY_EXISTSCATALOG_ALREADY_EXISTSSCHEMA_ALREADY_EXISTSTABLE_ALREADY_EXISTSSHARE_ALREADY_EXISTSRECIPIENT_ALREADY_EXISTSSTORAGE_CREDENTIAL_ALREADY_EXISTSEXTERNAL_LOCATION_ALREADY_EXISTSPROVIDER_ALREADY_EXISTSCATALOG_NOT_EMPTYSCHEMA_NOT_EMPTYMETASTORE_NOT_EMPTYPROVIDER_SHARE_NOT_ACCESSIBLE
- messagestring
- stack_tracestring
- detailsarray of object
- responseobject
The normal, successful response of the operation.