Synced Table
A synced table is a Postgres table in a Lakebase database instance that Databricks keeps in sync with a source table in Unity Catalog. A data synchronization pipeline loads the source data and then applies later changes according to the scheduling policy, so the table can serve low-latency queries without a hand-built ETL process. For an overview, see https://docs.databricks.com/aws/en/oltp/projects/sync-tables.
SyncedDatabaseTable object
A synced table is a Postgres table in a Lakebase database instance that Databricks keeps in sync with a source table in Unity Catalog. A data synchronization pipeline loads the source data and then applies later changes according to the scheduling policy, so the table can serve low-latency queries without a hand-built ETL process. For an overview, see https://docs.databricks.com/aws/en/oltp/projects/sync-tables.
- namestringPublic Preview
Full three-part (catalog, schema, table) name of the table.
- database_instance_namestringPublic PreviewInput only
Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).
- effective_database_instance_namestringPublic PreviewOutput only
The name of the database instance that this table is registered to. This field is always returned; for tables inside database catalogs it is the database instance associated with the catalog. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.
- logical_database_namestringPublic PreviewInput only
Target Postgres database object (logical database) name for this table.
When creating a synced table in a registered Postgres catalog, the target Postgres database name is inferred to be that of the registered catalog. If this field is specified in this scenario, the Postgres database name MUST match that of the registered catalog (or the request will be rejected).
When creating a synced table in a standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. If this database does not exist yet, it is created when
create_database_objects_if_missinginspecis true.
- effective_logical_database_namestringPublic PreviewOutput only
The name of the logical database that this table is registered to. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.
- specobjectPublic Preview
Specifies the source table, the key columns and how the data synchronization pipeline runs.
Show child attributesHide child attributes
- scheduling_policystringPublic Preview
Specifies the scheduling policy of the underlying pipeline. With CONTINUOUS, the pipeline keeps running and applies source changes as they happen. With TRIGGERED and SNAPSHOT, the pipeline stops after the initial load and syncs again only when it is run (manually, on a schedule or from a job); TRIGGERED applies source changes incrementally, while SNAPSHOT copies the full source table on every run. CONTINUOUS and TRIGGERED require the source table to provide a change data feed.
CONTINUOUSTRIGGEREDSNAPSHOT
- source_table_full_namestringPublic Preview
Three-part (catalog, schema, table) name of the source Delta table.
- primary_key_columnsarray of stringPublic Preview
The primary key columns are used to insert and update rows in the destination table. At least one column is required. Rows whose primary key columns contain null values are not synced, and the primary key cannot be changed after the table is created.
- timeseries_keystringPublic Preview
The time series key deduplicates (tie-breaks) rows with the same primary key. When primary key values can repeat in the source table, the synced table keeps only the row with the latest value of this column for each primary key. The key cannot be changed after the table is created.
- existing_pipeline_idstringPublic PreviewInput only
At most one of existing_pipeline_id and new_pipeline_spec should be defined.
If existing_pipeline_id is defined, the synced table will be bin packed into the existing pipeline referenced. This avoids creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline.
- create_database_objects_if_missingbooleanPublic PreviewInput only
If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request fails if this is false and the database or schema does not exist.
- new_pipeline_specobjectPublic PreviewInput only
At most one of existing_pipeline_id and new_pipeline_spec should be defined.
If new_pipeline_spec is defined, a new pipeline is created for this synced table. The location pointed to is used to store intermediate files (checkpoints, event logs etc). The caller must have write permissions to create Delta tables in the specified catalog and schema. Again, note this requires write permissions, whereas the source table only requires read permissions.
Show child attributesHide child attributes
- storage_catalogstringPublic Preview
This field needs to be specified if the destination catalog is a managed postgres catalog.
UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.
- storage_schemastringPublic Preview
This field needs to be specified if the destination catalog is a managed postgres catalog.
UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.
- budget_policy_idstringBeta
Budget policy to set on the newly created pipeline.
- unity_catalog_provisioning_statestringPublic PreviewOutput only
The provisioning state of the synced table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).
PROVISIONINGACTIVEFAILEDDELETINGUPDATINGDEGRADED
- data_synchronization_statusobjectPublic PreviewOutput only
Reports the status of the data synchronization pipeline that keeps this table in sync with its source, including the pipeline ID and the position of the last successful sync.
Show child attributesHide child attributes
- detailed_statestringPublic PreviewOutput only
The state of the synced table.
SYNCED_TABLE_PROVISIONINGSYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCESSYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOTSYNCED_TABLE_ONLINESYNCED_TABLE_ONLINE_CONTINUOUS_UPDATESYNCED_TABLE_ONLINE_TRIGGERED_UPDATESYNCED_TABLE_ONLINE_NO_PENDING_UPDATESYNCED_TABLED_OFFLINESYNCED_TABLE_OFFLINE_FAILEDSYNCED_TABLE_ONLINE_PIPELINE_FAILEDSYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES
- messagestringPublic PreviewOutput only
A text description of the current state of the synced table.
- provisioning_statusobjectPublic Preview
Provides detailed status while the table is being provisioned. It is populated when detailed_state is SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES or SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT.
Show child attributesHide child attributes
- initial_pipeline_sync_progressobjectPublic PreviewOutput only
Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.
Show child attributesHide child attributes
- latest_version_currently_processingint64Public PreviewOutput only
The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.
- synced_row_countint64Public PreviewOutput only
The number of rows that have been synced in this update.
- total_row_countint64Public PreviewOutput only
The total number of rows that need to be synced in this update. This number may be an estimate.
- sync_progress_completiondoublePublic PreviewOutput only
The completion ratio of this update. This is a number between 0 and 1.
- estimated_completion_time_secondsdoublePublic PreviewOutput only
The estimated time remaining to complete this update in seconds.
- provisioning_phasestringPublic PreviewOutput only
The current phase of the data synchronization pipeline.
PROVISIONING_PHASE_MAINPROVISIONING_PHASE_INDEX_SCANPROVISIONING_PHASE_INDEX_SORT
- continuous_update_statusobjectPublic Preview
Provides detailed status for an online table with the CONTINUOUS scheduling policy. It is populated when detailed_state is SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE or SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES.
Show child attributesHide child attributes
- last_processed_commit_versionint64Public PreviewOutput only
The last source table Delta version that was successfully synced to the synced table.
- timestampstringPublic PreviewOutput only
The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.
- initial_pipeline_sync_progressobjectPublic PreviewOutput only
Progress of the initial data synchronization.
Show child attributesHide child attributes
- latest_version_currently_processingint64Public PreviewOutput only
The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.
- synced_row_countint64Public PreviewOutput only
The number of rows that have been synced in this update.
- total_row_countint64Public PreviewOutput only
The total number of rows that need to be synced in this update. This number may be an estimate.
- sync_progress_completiondoublePublic PreviewOutput only
The completion ratio of this update. This is a number between 0 and 1.
- estimated_completion_time_secondsdoublePublic PreviewOutput only
The estimated time remaining to complete this update in seconds.
- provisioning_phasestringPublic PreviewOutput only
The current phase of the data synchronization pipeline.
PROVISIONING_PHASE_MAINPROVISIONING_PHASE_INDEX_SCANPROVISIONING_PHASE_INDEX_SORT
- triggered_update_statusobjectPublic Preview
Provides detailed status for an online table with the TRIGGERED or SNAPSHOT scheduling policy. It is populated when detailed_state is SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE or SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES.
Show child attributesHide child attributes
- last_processed_commit_versionint64Public PreviewOutput only
The last source table Delta version that was successfully synced to the synced table.
- timestampstringPublic PreviewOutput only
The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.
- triggered_update_progressobjectPublic PreviewOutput only
Progress of the active data synchronization pipeline.
Show child attributesHide child attributes
- latest_version_currently_processingint64Public PreviewOutput only
The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.
- synced_row_countint64Public PreviewOutput only
The number of rows that have been synced in this update.
- total_row_countint64Public PreviewOutput only
The total number of rows that need to be synced in this update. This number may be an estimate.
- sync_progress_completiondoublePublic PreviewOutput only
The completion ratio of this update. This is a number between 0 and 1.
- estimated_completion_time_secondsdoublePublic PreviewOutput only
The estimated time remaining to complete this update in seconds.
- provisioning_phasestringPublic PreviewOutput only
The current phase of the data synchronization pipeline.
PROVISIONING_PHASE_MAINPROVISIONING_PHASE_INDEX_SCANPROVISIONING_PHASE_INDEX_SORT
- failed_statusobjectPublic Preview
Provides detailed status for a table whose data synchronization pipeline has failed. It is populated when detailed_state is SYNCED_TABLE_OFFLINE_FAILED or SYNCED_TABLE_ONLINE_PIPELINE_FAILED.
Show child attributesHide child attributes
- last_processed_commit_versionint64Public PreviewOutput only
The last source table Delta version that was successfully synced to the synced table. Only populated if the table is still synced and available for serving.
- timestampstringPublic PreviewOutput only
The end timestamp of the last time any data was synchronized from the source table to the synced table. Only populated if the table is still synced and available for serving.
- pipeline_idstringPublic PreviewOutput only
ID of the associated pipeline. The pipeline ID may have been provided by the client (in the case of bin packing), or generated by the server (when creating a new pipeline).
- last_syncobjectPublic PreviewOutput only
Summary of the last successful synchronization from source to destination.
Will always be present if there has been a successful sync. Even if the most recent syncs have failed.
Limitation: The only exception is if the synced table is doing a FULL REFRESH, then the last sync information will not be available until the full refresh is complete. This limitation will be addressed in a future version.
This top-level field is a convenience for consumers who want easy access to last sync information without having to traverse detailed_status.
Show child attributesHide child attributes
- sync_start_timestampstringPublic PreviewOutput only
The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.
- sync_end_timestampstringPublic PreviewOutput only
The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.
- delta_table_sync_infoobjectPublic PreviewOutput only
Identifies the position in the source Delta table at the time of the last successful sync.
Show child attributesHide child attributes
- delta_commit_versionint64Public PreviewOutput only
The Delta Lake commit version that was last successfully synced.
- delta_commit_timestampstringPublic PreviewOutput only
The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.
Get a Synced Database Table Public Preview
GET
Get a Synced Database Table.
For new workloads, Databricks recommends the Postgres API equivalent: Get a synced table.
API scopes: postgres
Parameters
- namestringRequiredpath
Specifies the synced table by its full three-part (catalog, schema, table) name.
Response
Returns the SyncedDatabaseTable object.
Create a Synced Database Table Public Preview
POST
Create a Synced Database Table.
The table and its data synchronization pipeline are provisioned asynchronously. The response reflects the initial state; use Get a Synced Database Table to follow the data synchronization status until the table is online.
For new workloads, Databricks recommends the Postgres API equivalent: Create a synced table.
API scopes: postgres
Request body
- namestringRequired
Full three-part (catalog, schema, table) name of the table.
- database_instance_namestringInput only
Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).
- logical_database_namestringInput only
Target Postgres database object (logical database) name for this table.
When creating a synced table in a registered Postgres catalog, the target Postgres database name is inferred to be that of the registered catalog. If this field is specified in this scenario, the Postgres database name MUST match that of the registered catalog (or the request will be rejected).
When creating a synced table in a standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. If this database does not exist yet, it is created when
create_database_objects_if_missinginspecis true.
- specobject
Specifies the source table, the key columns and how the data synchronization pipeline runs.
Show child attributesHide child attributes
- scheduling_policystring
Specifies the scheduling policy of the underlying pipeline. With CONTINUOUS, the pipeline keeps running and applies source changes as they happen. With TRIGGERED and SNAPSHOT, the pipeline stops after the initial load and syncs again only when it is run (manually, on a schedule or from a job); TRIGGERED applies source changes incrementally, while SNAPSHOT copies the full source table on every run. CONTINUOUS and TRIGGERED require the source table to provide a change data feed.
CONTINUOUSTRIGGEREDSNAPSHOT
- source_table_full_namestring
Three-part (catalog, schema, table) name of the source Delta table.
- primary_key_columnsarray of string
The primary key columns are used to insert and update rows in the destination table. At least one column is required. Rows whose primary key columns contain null values are not synced, and the primary key cannot be changed after the table is created.
- timeseries_keystring
The time series key deduplicates (tie-breaks) rows with the same primary key. When primary key values can repeat in the source table, the synced table keeps only the row with the latest value of this column for each primary key. The key cannot be changed after the table is created.
- existing_pipeline_idstringInput only
At most one of existing_pipeline_id and new_pipeline_spec should be defined.
If existing_pipeline_id is defined, the synced table will be bin packed into the existing pipeline referenced. This avoids creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline.
- create_database_objects_if_missingbooleanInput only
If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request fails if this is false and the database or schema does not exist.
- new_pipeline_specobjectInput only
At most one of existing_pipeline_id and new_pipeline_spec should be defined.
If new_pipeline_spec is defined, a new pipeline is created for this synced table. The location pointed to is used to store intermediate files (checkpoints, event logs etc). The caller must have write permissions to create Delta tables in the specified catalog and schema. Again, note this requires write permissions, whereas the source table only requires read permissions.
Show child attributesHide child attributes
- storage_catalogstring
This field needs to be specified if the destination catalog is a managed postgres catalog.
UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.
- storage_schemastring
This field needs to be specified if the destination catalog is a managed postgres catalog.
UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.
- budget_policy_idstringBeta
Budget policy to set on the newly created pipeline.
Response
Returns the SyncedDatabaseTable object.
Delete a Synced Database Table Public Preview
DELETE
Delete a Synced Database Table.
For new workloads, Databricks recommends the Postgres API equivalent: Delete a synced table.
API scopes: postgres
Parameters
- namestringRequiredpath
Specifies the synced table by its full three-part (catalog, schema, table) name.
- purge_databooleanquery
When true, the Postgres table is also dropped from the database. When false, the synced table is unregistered from Unity Catalog and its pipeline stops syncing it, but the Postgres table and its data are kept.