create_streaming_table

Use the create_streaming_table() function in a pipeline to create a target table for records output by streaming operations, including create_auto_cdc_flow(), create_auto_cdc_from_snapshot_flow(), and append_flow output records.

note

The create_target_table() and create_streaming_live_table() functions are deprecated. Databricks recommends updating existing code to use the create_streaming_table() function.

Syntax

Python

from pyspark import pipelines as dp

dp.create_streaming_table(
  name = "<table-name>",
  comment = "<comment>",
  spark_conf={"<key>" : "<value", "<key" : "<value>"},
  table_properties={"<key>" : "<value>", "<key>" : "<value>"},
  path="<storage-location-path>",
  partition_cols=["<partition-column>", "<partition-column>"],
  cluster_by_auto = False,
  cluster_by = ["<clustering-column>", "<clustering-column>"],
  schema="schema-definition",
  expect_all = {"<key>" : "<value", "<key" : "<value>"},
  expect_all_or_drop = {"<key>" : "<value", "<key" : "<value>"},
  expect_all_or_fail = {"<key>" : "<value", "<key" : "<value>"},
  row_filter = "row-filter-clause",
  private = False
)

Parameters

Parameter	Type	Description
name	str	Required. The table name.
comment	str	A description for the table.
spark_conf	dict	A list of Spark configurations for the execution of this query
table_properties	dict	A dict of table properties for the table.
path	str	A storage location for table data. If not set, use the managed storage location for the schema containing the table.
partition_cols	list	A list of one or more columns to use for partitioning the table.
cluster_by_auto	bool	Enable automatic liquid clustering on the table. This can be combined with cluster_by and define the columns to be use as initial clustering keys, followed by monitoring and automatic key selection updates based on the workload. See Automatic liquid clustering.
cluster_by	list	Enable liquid clustering on the table and define the columns to use as clustering keys. See Use liquid clustering for tables.
schema	str or StructType	A schema definition for the table. Schemas can be defined as a SQL DDL string or with a Python StructType.
expect_all, expect_all_or_drop, expect_all_or_fail	dict	Data quality constraints for the table. Provides the same behavior and uses the same syntax as expectation decorator functions, but implemented as a parameter. See Expectations.
row_filter	str	(Public Preview) A row filter clause for the table. See Publish tables with row filters and column masks.
private	bool	When True, creates a private streaming table that is not published to the catalog and is only accessible within the pipeline. Private streaming tables persist for the lifetime of the pipeline. Private streaming tables were previously created with the temporary parameter. Defaults to False.

Parameter	Type	Description
name	str	Required. The table name.
comment	str	A description for the table.
spark_conf	dict	A list of Spark configurations for the execution of this query
table_properties	dict	A dict of table properties for the table.
path	str	A storage location for table data. If not set, use the managed storage location for the schema containing the table.
partition_cols	list	A list of one or more columns to use for partitioning the table.
cluster_by_auto	bool	Enable automatic liquid clustering on the table. This can be combined with cluster_by and define the columns to be use as initial clustering keys, followed by monitoring and automatic key selection updates based on the workload. See Automatic liquid clustering.
cluster_by	list	Enable liquid clustering on the table and define the columns to use as clustering keys. See Use liquid clustering for tables.
schema	str or StructType	A schema definition for the table. Schemas can be defined as a SQL DDL string or with a Python StructType.
expect_all, expect_all_or_drop, expect_all_or_fail	dict	Data quality constraints for the table. Provides the same behavior and uses the same syntax as expectation decorator functions, but implemented as a parameter. See Expectations.
row_filter	str	(Public Preview) A row filter clause for the table. See Publish tables with row filters and column masks.
private	bool	When True, creates a private streaming table that is not published to the catalog and is only accessible within the pipeline. Private streaming tables persist for the lifetime of the pipeline. Private streaming tables were previously created with the temporary parameter. Defaults to False.