Skip to main content

Synced Table

View as Markdown

A synced table is a Postgres table in a Lakebase database instance that Databricks keeps in sync with a source table in Unity Catalog. A data synchronization pipeline loads the source data and then applies later changes according to the scheduling policy, so the table can serve low-latency queries without a hand-built ETL process. For an overview, see https://docs.databricks.com/aws/en/oltp/projects/sync-tables.

SyncedDatabaseTable object​

A synced table is a Postgres table in a Lakebase database instance that Databricks keeps in sync with a source table in Unity Catalog. A data synchronization pipeline loads the source data and then applies later changes according to the scheduling policy, so the table can serve low-latency queries without a hand-built ETL process. For an overview, see https://docs.databricks.com/aws/en/oltp/projects/sync-tables.

namestringPublic Preview

Full three-part (catalog, schema, table) name of the table.

database_instance_namestringPublic PreviewInput only

Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).

effective_database_instance_namestringPublic PreviewOutput only

The name of the database instance that this table is registered to. This field is always returned; for tables inside database catalogs it is the database instance associated with the catalog. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

logical_database_namestringPublic PreviewInput only

Target Postgres database object (logical database) name for this table.

When creating a synced table in a registered Postgres catalog, the target Postgres database name is inferred to be that of the registered catalog. If this field is specified in this scenario, the Postgres database name MUST match that of the registered catalog (or the request will be rejected).

When creating a synced table in a standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. If this database does not exist yet, it is created when create_database_objects_if_missing in spec is true.

effective_logical_database_namestringPublic PreviewOutput only

The name of the logical database that this table is registered to. This is an output only field that contains the value computed from the input field combined with server side defaults. Use the field without the effective_ prefix to set the value.

specobjectPublic Preview

Specifies the source table, the key columns and how the data synchronization pipeline runs.

Show child attributesHide child attributes
scheduling_policystringPublic Preview

Specifies the scheduling policy of the underlying pipeline. With CONTINUOUS, the pipeline keeps running and applies source changes as they happen. With TRIGGERED and SNAPSHOT, the pipeline stops after the initial load and syncs again only when it is run (manually, on a schedule or from a job); TRIGGERED applies source changes incrementally, while SNAPSHOT copies the full source table on every run. CONTINUOUS and TRIGGERED require the source table to provide a change data feed.

Values:

  • CONTINUOUS
  • TRIGGERED
  • SNAPSHOT
source_table_full_namestringPublic Preview

Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of stringPublic Preview

The primary key columns are used to insert and update rows in the destination table. At least one column is required. Rows whose primary key columns contain null values are not synced, and the primary key cannot be changed after the table is created.

timeseries_keystringPublic Preview

The time series key deduplicates (tie-breaks) rows with the same primary key. When primary key values can repeat in the source table, the synced table keeps only the row with the latest value of this column for each primary key. The key cannot be changed after the table is created.

existing_pipeline_idstringPublic PreviewInput only

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If existing_pipeline_id is defined, the synced table will be bin packed into the existing pipeline referenced. This avoids creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline.

create_database_objects_if_missingbooleanPublic PreviewInput only

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request fails if this is false and the database or schema does not exist.

Default: true

new_pipeline_specobjectPublic PreviewInput only

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If new_pipeline_spec is defined, a new pipeline is created for this synced table. The location pointed to is used to store intermediate files (checkpoints, event logs etc). The caller must have write permissions to create Delta tables in the specified catalog and schema. Again, note this requires write permissions, whereas the source table only requires read permissions.

Show child attributesHide child attributes
storage_catalogstringPublic Preview

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastringPublic Preview

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstringBeta

Budget policy to set on the newly created pipeline.

unity_catalog_provisioning_statestringPublic PreviewOutput only

The provisioning state of the synced table entity in Unity Catalog. This is distinct from the state of the data synchronization pipeline (i.e. the table may be in "ACTIVE" but the pipeline may be in "PROVISIONING" as it runs asynchronously).

Values:

  • PROVISIONING
  • ACTIVE
  • FAILED
  • DELETING
  • UPDATING
  • DEGRADED
data_synchronization_statusobjectPublic PreviewOutput only

Reports the status of the data synchronization pipeline that keeps this table in sync with its source, including the pipeline ID and the position of the last successful sync.

Show child attributesHide child attributes
detailed_statestringPublic PreviewOutput only

The state of the synced table.

Values:

  • SYNCED_TABLE_PROVISIONING
  • SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES
  • SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT
  • SYNCED_TABLE_ONLINE
  • SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE
  • SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE
  • SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE
  • SYNCED_TABLED_OFFLINE
  • SYNCED_TABLE_OFFLINE_FAILED
  • SYNCED_TABLE_ONLINE_PIPELINE_FAILED
  • SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES
messagestringPublic PreviewOutput only

A text description of the current state of the synced table.

provisioning_statusobjectPublic Preview

Provides detailed status while the table is being provisioned. It is populated when detailed_state is SYNCED_TABLE_PROVISIONING_PIPELINE_RESOURCES or SYNCED_TABLE_PROVISIONING_INITIAL_SNAPSHOT.

Show child attributesHide child attributes
initial_pipeline_sync_progressobjectPublic PreviewOutput only

Details about initial data synchronization. Only populated when in the PROVISIONING_INITIAL_SNAPSHOT state.

Show child attributesHide child attributes
latest_version_currently_processingint64Public PreviewOutput only

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public PreviewOutput only

The number of rows that have been synced in this update.

total_row_countint64Public PreviewOutput only

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic PreviewOutput only

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic PreviewOutput only

The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic PreviewOutput only

The current phase of the data synchronization pipeline.

Values:

  • PROVISIONING_PHASE_MAIN
  • PROVISIONING_PHASE_INDEX_SCAN
  • PROVISIONING_PHASE_INDEX_SORT
continuous_update_statusobjectPublic Preview

Provides detailed status for an online table with the CONTINUOUS scheduling policy. It is populated when detailed_state is SYNCED_TABLE_ONLINE_CONTINUOUS_UPDATE or SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES.

Show child attributesHide child attributes
last_processed_commit_versionint64Public PreviewOutput only

The last source table Delta version that was successfully synced to the synced table.

timestampstringPublic PreviewOutput only

The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

initial_pipeline_sync_progressobjectPublic PreviewOutput only

Progress of the initial data synchronization.

Show child attributesHide child attributes
latest_version_currently_processingint64Public PreviewOutput only

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public PreviewOutput only

The number of rows that have been synced in this update.

total_row_countint64Public PreviewOutput only

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic PreviewOutput only

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic PreviewOutput only

The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic PreviewOutput only

The current phase of the data synchronization pipeline.

Values:

  • PROVISIONING_PHASE_MAIN
  • PROVISIONING_PHASE_INDEX_SCAN
  • PROVISIONING_PHASE_INDEX_SORT
triggered_update_statusobjectPublic Preview

Provides detailed status for an online table with the TRIGGERED or SNAPSHOT scheduling policy. It is populated when detailed_state is SYNCED_TABLE_ONLINE_TRIGGERED_UPDATE, SYNCED_TABLE_ONLINE_NO_PENDING_UPDATE or SYNCED_TABLE_ONLINE_UPDATING_PIPELINE_RESOURCES.

Show child attributesHide child attributes
last_processed_commit_versionint64Public PreviewOutput only

The last source table Delta version that was successfully synced to the synced table.

timestampstringPublic PreviewOutput only

The end timestamp of the last time any data was synchronized from the source table to the synced table. This is when the data is available in the synced table.

triggered_update_progressobjectPublic PreviewOutput only

Progress of the active data synchronization pipeline.

Show child attributesHide child attributes
latest_version_currently_processingint64Public PreviewOutput only

The source table Delta version that was last processed by the pipeline. The pipeline may not have completely processed this version yet.

synced_row_countint64Public PreviewOutput only

The number of rows that have been synced in this update.

total_row_countint64Public PreviewOutput only

The total number of rows that need to be synced in this update. This number may be an estimate.

sync_progress_completiondoublePublic PreviewOutput only

The completion ratio of this update. This is a number between 0 and 1.

estimated_completion_time_secondsdoublePublic PreviewOutput only

The estimated time remaining to complete this update in seconds.

provisioning_phasestringPublic PreviewOutput only

The current phase of the data synchronization pipeline.

Values:

  • PROVISIONING_PHASE_MAIN
  • PROVISIONING_PHASE_INDEX_SCAN
  • PROVISIONING_PHASE_INDEX_SORT
failed_statusobjectPublic Preview

Provides detailed status for a table whose data synchronization pipeline has failed. It is populated when detailed_state is SYNCED_TABLE_OFFLINE_FAILED or SYNCED_TABLE_ONLINE_PIPELINE_FAILED.

Show child attributesHide child attributes
last_processed_commit_versionint64Public PreviewOutput only

The last source table Delta version that was successfully synced to the synced table. Only populated if the table is still synced and available for serving.

timestampstringPublic PreviewOutput only

The end timestamp of the last time any data was synchronized from the source table to the synced table. Only populated if the table is still synced and available for serving.

pipeline_idstringPublic PreviewOutput only

ID of the associated pipeline. The pipeline ID may have been provided by the client (in the case of bin packing), or generated by the server (when creating a new pipeline).

last_syncobjectPublic PreviewOutput only

Summary of the last successful synchronization from source to destination.

Will always be present if there has been a successful sync. Even if the most recent syncs have failed.

Limitation: The only exception is if the synced table is doing a FULL REFRESH, then the last sync information will not be available until the full refresh is complete. This limitation will be addressed in a future version.

This top-level field is a convenience for consumers who want easy access to last sync information without having to traverse detailed_status.

Show child attributesHide child attributes
sync_start_timestampstringPublic PreviewOutput only

The starting timestamp of the most recent successful synchronization from the source table to the destination (synced) table. Note this is the starting timestamp of the sync operation, not the end time. E.g., for a batch, this is the time when the sync operation started.

sync_end_timestampstringPublic PreviewOutput only

The end timestamp of the most recent successful synchronization. This is the time when the data is available in the synced table.

delta_table_sync_infoobjectPublic PreviewOutput only

Identifies the position in the source Delta table at the time of the last successful sync.

Show child attributesHide child attributes
delta_commit_versionint64Public PreviewOutput only

The Delta Lake commit version that was last successfully synced.

delta_commit_timestampstringPublic PreviewOutput only

The timestamp when the above Delta version was committed in the source Delta table. Note: This is the Delta commit time, not the time the data was written to the synced table.

Get a Synced Database Table Public Preview​

GET /api/2.0/database/synced_tables/{name}

Get a Synced Database Table.

For new workloads, Databricks recommends the Postgres API equivalent: Get a synced table.

API scopes: postgres

CloudsAWSAzure

Parameters​

namestringRequiredpath

Specifies the synced table by its full three-part (catalog, schema, table) name.

Response​

Returns the SyncedDatabaseTable object.

Create a Synced Database Table Public Preview​

POST /api/2.0/database/synced_tables

Create a Synced Database Table.

The table and its data synchronization pipeline are provisioned asynchronously. The response reflects the initial state; use Get a Synced Database Table to follow the data synchronization status until the table is online.

For new workloads, Databricks recommends the Postgres API equivalent: Create a synced table.

API scopes: postgres

CloudsAWSAzure

Request body​

namestringRequired

Full three-part (catalog, schema, table) name of the table.

database_instance_namestringInput only

Name of the target database instance. This is required when creating synced database tables in standard catalogs. This is optional when creating synced database tables in registered catalogs. If this field is specified when creating synced database tables in registered catalogs, the database instance name MUST match that of the registered catalog (or the request will be rejected).

logical_database_namestringInput only

Target Postgres database object (logical database) name for this table.

When creating a synced table in a registered Postgres catalog, the target Postgres database name is inferred to be that of the registered catalog. If this field is specified in this scenario, the Postgres database name MUST match that of the registered catalog (or the request will be rejected).

When creating a synced table in a standard catalog, this field is required. In this scenario, specifying this field will allow targeting an arbitrary postgres database. If this database does not exist yet, it is created when create_database_objects_if_missing in spec is true.

specobject

Specifies the source table, the key columns and how the data synchronization pipeline runs.

Show child attributesHide child attributes
scheduling_policystring

Specifies the scheduling policy of the underlying pipeline. With CONTINUOUS, the pipeline keeps running and applies source changes as they happen. With TRIGGERED and SNAPSHOT, the pipeline stops after the initial load and syncs again only when it is run (manually, on a schedule or from a job); TRIGGERED applies source changes incrementally, while SNAPSHOT copies the full source table on every run. CONTINUOUS and TRIGGERED require the source table to provide a change data feed.

Values:

  • CONTINUOUS
  • TRIGGERED
  • SNAPSHOT
source_table_full_namestring

Three-part (catalog, schema, table) name of the source Delta table.

primary_key_columnsarray of string

The primary key columns are used to insert and update rows in the destination table. At least one column is required. Rows whose primary key columns contain null values are not synced, and the primary key cannot be changed after the table is created.

timeseries_keystring

The time series key deduplicates (tie-breaks) rows with the same primary key. When primary key values can repeat in the source table, the synced table keeps only the row with the latest value of this column for each primary key. The key cannot be changed after the table is created.

existing_pipeline_idstringInput only

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If existing_pipeline_id is defined, the synced table will be bin packed into the existing pipeline referenced. This avoids creating a new pipeline and allows sharing existing compute. In this case, the scheduling_policy of this synced table must match the scheduling policy of the existing pipeline.

create_database_objects_if_missingbooleanInput only

If true, the synced table's logical database and schema resources in PG will be created if they do not already exist. The request fails if this is false and the database or schema does not exist.

Default: true

new_pipeline_specobjectInput only

At most one of existing_pipeline_id and new_pipeline_spec should be defined.

If new_pipeline_spec is defined, a new pipeline is created for this synced table. The location pointed to is used to store intermediate files (checkpoints, event logs etc). The caller must have write permissions to create Delta tables in the specified catalog and schema. Again, note this requires write permissions, whereas the source table only requires read permissions.

Show child attributesHide child attributes
storage_catalogstring

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC catalog for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be a standard catalog where the user has permissions to create Delta tables.

storage_schemastring

This field needs to be specified if the destination catalog is a managed postgres catalog.

UC schema for the pipeline to store intermediate files (checkpoints, event logs etc). This needs to be in the standard catalog where the user has permissions to create Delta tables.

budget_policy_idstringBeta

Budget policy to set on the newly created pipeline.

Response​

Returns the SyncedDatabaseTable object.

Delete a Synced Database Table Public Preview​

DELETE /api/2.0/database/synced_tables/{name}

Delete a Synced Database Table.

For new workloads, Databricks recommends the Postgres API equivalent: Delete a synced table.

API scopes: postgres

CloudsAWSAzure

Parameters​

namestringRequiredpath

Specifies the synced table by its full three-part (catalog, schema, table) name.

purge_databooleanquery

When true, the Postgres table is also dropped from the database. When false, the synced table is unregistered from Unity Catalog and its pipeline stops syncing it, but the Postgres table and its data are kept.