Pipeline

View as Markdown

PipelineSpec object

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

Nodes for this cluster will only be placed on AWS instances with this instance profile. If omitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogeneously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

NOTE: Each instance type can only support a certain number of attached local SSDs. The value specified in local_ssd_count must be valid for BOTH the driver and worker instance type. See GCP docs here: https://cloud.google.com/compute/docs/disks#local_ssd_machine_type_restrictions

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

Show child attributesHide child attributes

dbfsobject

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

Show child attributesHide child attributes

dbfsobject

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3object

destination and either region or endpoint should also be provided. e.g. { "s3": { "destination" : "s3://init-scripts/my_script.sh", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: Flag to enable server side encryption, false by default.

encryption_typestring: The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobject

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstring: URI of the whl to be installed.

notebookobject

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobject

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectPublic Preview

Select all tables from a specific source schema.

Show child attributesHide child attributes

source_catalogstringPublic Preview: The source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store tables.

destination_schemastringPublic Preview: Required. Destination schema to store tables in. Tables with the same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in this schema and override the table_configuration defined in the IngestionPipelineDefinition object.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

fanout_optionsobjectBeta: Fanout options for multi-table routing from streaming sources. When set, records are routed to destination tables based on a per-record routing key. The key value becomes the table name: {destination_catalog}.{destination_schema}.{key_value}.

tableobjectPublic Preview

Select a specific source table.

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Might be optional depending on the type of source.

source_tablestringPublic Preview: Table name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Optional. Destination table name. The pipeline fails if a table with that name already exists. If not set, the source table name is used.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object and the SchemaSpec.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

reportobjectPublic Preview

Select a specific source report.

Show child attributesHide child attributes

source_urlstringPublic Preview: Required. Report URL in the source system.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Required. Destination table name. The pipeline fails if a table with that name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

Show child attributesHide child attributes

cursor_columnsarray of stringPublic Preview: The names of the monotonically increasing columns in the source table that are used to enable the table to be read and ingested incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the sequence_by behavior. You can still explicitly set sequence_by to override this default.

deletion_conditionstringPublic Preview: Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to hard_deletion_sync_interval_in_seconds, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table.

hard_deletion_sync_min_interval_in_secondsint64Beta: Specifies the minimum interval (in seconds) between snapshots on primary keys for detecting and synchronizing hard deletions—i.e., rows that have been physically removed from the source table. This interval acts as a lower bound. If ingestion runs less frequently than this value, hard deletion synchronization will align with the actual ingestion frequency instead of happening more often. If not set, hard deletion synchronization via snapshots is disabled. This field is mutable and can be updated without triggering a full snapshot.

auto_full_refresh_policyobjectPublic Preview

(Optional, Mutable) Policy for auto full refresh, if enabled pipeline will automatically try to fix issues by doing a full refresh on the table in the retry run. auto_full_refresh_policy in table configuration will override the above level auto_full_refresh_policy. For example, { "auto_full_refresh_policy": { "enabled": true, "min_interval_hours": 23, } } If unspecified, auto full refresh is disabled.

Show child attributesHide child attributes

enabledbooleanPublic Preview: (Required, Mutable) Whether to enable auto full refresh or not.

min_interval_hoursint32Public Preview: (Optional, Mutable) Specify the minimum interval in hours between the timestamp at which a table was last full refreshed and the current timestamp for triggering auto full If unspecified and autoFullRefresh is enabled then by default min_interval_hours is 24 hours.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview

Catalog-level source configuration parameters

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name

postgresobjectPublic Preview: Postgres-specific catalog-level configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

(Optional) Location of staged data storage. This is required for migration from Cdc Managed Ingestion Pipeline with Gateway pipeline to Combined Cdc Managed Ingestion Pipeline. If not specified, the volume for staged data will be created in catalog and schema/target specified in the top level pipeline definition.

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobject

cronobject

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

Deprecated: wrap the pipeline in a continuous job instead, which also lets you take advantage of job-level settings such as performance mode. When the pipeline is started by a continuous job, the job's setting takes precedence and this field is ignored.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobjectGA: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The environment version of the serverless Python environment used to execute customer Python code. Each environment version includes a specific Python version and a curated set of pre-installed libraries with defined versions, providing a stable and reproducible execution environment.

<Databricks> supports a three-year lifecycle for each environment version. For available versions and their included packages, see https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/

The value should be a string representing the environment version number, for example: "4".

View as Markdown

The PipelineSpec object

{
  "id": "string",
  "name": "string",
  "storage": "string",
  "configuration": {},
  "clusters": [
    {
      "label": "string",
      "apply_policy_default_values": true,
      "spark_conf": {
        "spark.speculation": true,
        "spark.streaming.ui.retainedBatches": 5
      },
      "aws_attributes": {},
      "azure_attributes": {},
      "gcp_attributes": {},
      "node_type_id": "string",
      "driver_node_type_id": "string",
      "ssh_public_keys": [
        "string"
      ],
      "custom_tags": {},
      "cluster_log_conf": {},
      "spark_env_vars": {},
      "init_scripts": [
        {}
      ],
      "instance_pool_id": "string",
      "policy_id": "string",
      "enable_local_disk_encryption": true,
      "driver_instance_pool_id": "string",
      "num_workers": 0,
      "autoscale": {}
    }
  ],
  "libraries": [
    {
      "whl": "string",
      "notebook": {
        "path": "/my-pipeline-notebook-path"
      },
      "file": {
        "path": "/Repos/user@databricks.com/my_repo/my_code.py"
      },
      "glob": {}
    }
  ],
  "ingestion_definition": {
    "connection_name": "string",
    "ingestion_gateway_id": "string",
    "ingest_from_uc_foreign_catalog": true,
    "objects": [
      {}
    ],
    "source_type": "string",
    "table_configuration": {},
    "source_configurations": [
      {}
    ],
    "full_refresh_window": {},
    "connector_type": "string",
    "data_staging_options": {}
  },
  "trigger": {
    "manual": {},
    "cron": {}
  },
  "target": "string",
  "schema": "string",
  "filters": {
    "include": [
      "string"
    ],
    "exclude": [
      "string"
    ]
  },
  "continuous": true,
  "development": true,
  "photon": true,
  "edition": "string",
  "channel": "string",
  "catalog": "string",
  "notifications": [
    {
      "email_recipients": [
        "string"
      ],
      "alerts": [
        "string"
      ]
    }
  ],
  "serverless": true,
  "deployment": {
    "kind": "string",
    "metadata_file_path": "string"
  },
  "budget_policy_id": "string",
  "tags": {},
  "event_log": {
    "name": "string",
    "schema": "string",
    "catalog": "string"
  },
  "root_path": "/Workspace/Users/user.name/my-pipeline",
  "environment": {
    "dependencies": [
      "foo==0.0.1",
      "-r /Workspace/Users/user.name/my-pipeline/requirements.txt"
    ],
    "environment_version": "string"
  }
}

Get GA

GET /api/2.0/pipelines/{pipeline_id}

Get a pipeline.

API scopes: pipelines

Parameters

pipeline_idstringpath

Response

pipeline_idstring: The ID of the pipeline.

specobject

The pipeline specification. This field is not returned when called by ListPipelines.

Show child attributesHide child attributes

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

spark_env_varsobject

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

s3objectRequired: destination and either region or endpoint should also be provided. e.g. { "s3": { "destination" : "s3://init-scripts/my_script.sh", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobjectRequired

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectRequiredPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectRequiredPublic Preview: Select all tables from a specific source schema.

tableobjectRequiredPublic Preview: Select a specific source table.

reportobjectRequiredPublic Preview: Select a specific source report.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

auto_full_refresh_policyobjectPublic Preview: (Optional, Mutable) Policy for auto full refresh, if enabled pipeline will automatically try to fix issues by doing a full refresh on the table in the retry run. auto_full_refresh_policy in table configuration will override the above level auto_full_refresh_policy. For example, { "auto_full_refresh_policy": { "enabled": true, "min_interval_hours": 23, } } If unspecified, auto full refresh is disabled.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview: Catalog-level source configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

statestring: The pipeline state.
Values: DEPLOYING, STARTING, RUNNING, STOPPING, DELETED, RECOVERING, FAILED, RESETTING, IDLE

causestring: An optional message detailing the cause of the pipeline state.

cluster_idstring: The ID of the cluster that the pipeline is running on.

namestring: A human friendly identifier for the pipeline, taken from the spec.

healthstring: The health of a pipeline.
Values: HEALTHY, UNHEALTHY

creator_user_namestring: The username of the pipeline creator.

latest_updatesarray of object

Status of the latest updates for the pipeline. Ordered with the newest update first.

Show child attributesHide child attributes

update_idstring

statestring: Values: QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, CANCELED

creation_timestring

last_modifiedint64: The last time the pipeline settings were modified or created.

run_as_user_namestring: Username of the user that the pipeline will run on behalf of.

effective_publishing_modestring: Publishing mode of the pipeline
Values: PUBLISHING_MODE_UNSPECIFIED, LEGACY_PUBLISHING_MODE, DEFAULT_PUBLISHING_MODE

run_asobject

The user or service principal that the pipeline runs as, if specified in the request. This field indicates the explicit configuration of run_as for the pipeline. To find the value in all cases, explicit or implicit, use run_as_user_name.

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

parametersobjectBeta: Key/value map of default parameters to use for pipeline execution. Maximum total size: 10k characters (JSON format)

effective_serverless_compute_idstringBeta: Serverless compute ID resolved for the pipeline.

View as Markdown

Response

{
  "cluster_id": "1234-567891-abcde123",
  "creator_user_name": "username",
  "latest_updates": [
    {
      "creation_time": "2021-08-13T00:37:30.279Z",
      "state": "COMPLETED",
      "update_id": "8a0b6d02-fbd0-11eb-9a03-0242ac130003"
    },
    {
      "creation_time": "2021-08-13T00:35:51.902Z",
      "state": "CANCELED",
      "update_id": "a72c08ba-fbd0-11eb-9a03-0242ac130003"
    },
    {
      "creation_time": "2021-08-13T00:33:38.565Z",
      "state": "FAILED",
      "update_id": "ac37d924-fbd0-11eb-9a03-0242ac130003"
    }
  ],
  "name": "Wikipedia pipeline (SQL)",
  "pipeline_id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5",
  "run_as_user_name": "username",
  "spec": {
    "clusters": [
      {
        "autoscale": {
          "max_workers": 5,
          "min_workers": 1,
          "mode": "ENHANCED"
        },
        "label": "default"
      }
    ],
    "continuous": false,
    "deployment": {
      "kind": "BUNDLE",
      "metadata_file_path": "/Users/username/my_project/databricks.yml"
    },
    "id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5",
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/SDP Notebooks/Spark Declarative Pipelines quickstart (SQL)"
        }
      }
    ],
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "target": "wikipedia_quickstart_data"
  },
  "state": "IDLE"
}

List GA

GET /api/2.0/pipelines

Lists pipelines defined in the Spark Declarative Pipelines system.

API scopes: pipelines

Parameters

page_tokenstringquery: Page token returned by previous call

max_resultsint32query: The maximum number of entries to return in a single page. The system may return fewer than max_results events in a response, even if there are more events available. This field is optional. The default value is 25. The maximum value is 100. An error is returned if the value of max_results is greater than 100.
Default: 25
Constraints: <= 100

order_byarray of stringquery: A list of strings specifying the order of results. Supported order_by fields are id and name. The default is id asc. This field is optional.

filterstringquery

Select a subset of results based on the specified criteria. The supported filters are:

notebook='<path>' to select pipelines that reference the provided notebook path.
name LIKE '[pattern]' to select pipelines with a name that matches pattern. Wildcards are supported, for example: name LIKE '%shopping%'

Composite filters are not supported. This field is optional.

Response

statusesarray of object

The list of events matching the request criteria.

Show child attributesHide child attributes

pipeline_idstring: The unique identifier of the pipeline.

statestring: Values: DEPLOYING, STARTING, RUNNING, STOPPING, DELETED, RECOVERING, FAILED, RESETTING, IDLE

cluster_idstring: The unique identifier of the cluster running the pipeline.

namestring: The user-friendly name of the pipeline.

latest_updatesarray of object

Status of the latest updates for the pipeline. Ordered with the newest update first.

Show child attributesHide child attributes

update_idstring

statestring: Values: QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, CANCELED

creation_timestring

creator_user_namestring: The username of the pipeline creator.

run_as_user_namestring: The username that the pipeline runs as. This is a read only value derived from the pipeline owner.

healthstring: The health of a pipeline.
Values: HEALTHY, UNHEALTHY

next_page_tokenstring: If present, a token to fetch the next page of events.

View as Markdown

Response

{
  "next_page_token": "eyJ...==",
  "prev_page_token": "eyJ..x9",
  "statuses": [
    {
      "creator_user_name": "username",
      "latest_updates": [
        {
          "creation_time": "2021-08-13T00:34:21.871Z",
          "state": "COMPLETED",
          "update_id": "ee9ae73e-fc61-11eb-9a03-0242ac130003"
        }
      ],
      "name": "SDP quickstart (Python)",
      "pipeline_id": "e0f01758-fc61-11eb-9a03-0242ac130003",
      "state": "IDLE"
    },
    {
      "creator_user_name": "username",
      "name": "My SDP quickstart example",
      "pipeline_id": "f4c82f5e-fc61-11eb-9a03-0242ac130003",
      "state": "IDLE"
    }
  ]
}

Create GA

POST /api/2.0/pipelines

Creates a new data processing pipeline based on the requested configuration. If successful, this method returns the ID of the new pipeline.

API scopes: pipelines

Request body

allow_duplicate_namesboolean: If false, deployment will fail if name conflicts with that of another pipeline.
Default: false

dry_runboolean

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

parametersobjectBeta: Key/value map of default parameters to use for pipeline execution. Maximum total size: 10k characters (JSON format)

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

spark_env_varsobject

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: Flag to enable server side encryption, false by default.

encryption_typestring: The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobjectRequired

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectRequiredPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectRequiredPublic Preview

Select all tables from a specific source schema.

Show child attributesHide child attributes

source_catalogstringPublic Preview: The source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store tables.

destination_schemastringPublic Preview: Required. Destination schema to store tables in. Tables with the same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in this schema and override the table_configuration defined in the IngestionPipelineDefinition object.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

fanout_optionsobjectBeta: Fanout options for multi-table routing from streaming sources. When set, records are routed to destination tables based on a per-record routing key. The key value becomes the table name: {destination_catalog}.{destination_schema}.{key_value}.

tableobjectRequiredPublic Preview

Select a specific source table.

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Might be optional depending on the type of source.

source_tablestringPublic Preview: Table name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Optional. Destination table name. The pipeline fails if a table with that name already exists. If not set, the source table name is used.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object and the SchemaSpec.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

reportobjectRequiredPublic Preview

Select a specific source report.

Show child attributesHide child attributes

source_urlstringPublic Preview: Required. Report URL in the source system.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Required. Destination table name. The pipeline fails if a table with that name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

Show child attributesHide child attributes

cursor_columnsarray of stringPublic Preview: The names of the monotonically increasing columns in the source table that are used to enable the table to be read and ingested incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the sequence_by behavior. You can still explicitly set sequence_by to override this default.

deletion_conditionstringPublic Preview: Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to hard_deletion_sync_interval_in_seconds, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table.

hard_deletion_sync_min_interval_in_secondsint64Beta: Specifies the minimum interval (in seconds) between snapshots on primary keys for detecting and synchronizing hard deletions—i.e., rows that have been physically removed from the source table. This interval acts as a lower bound. If ingestion runs less frequently than this value, hard deletion synchronization will align with the actual ingestion frequency instead of happening more often. If not set, hard deletion synchronization via snapshots is disabled. This field is mutable and can be updated without triggering a full snapshot.

auto_full_refresh_policyobjectPublic Preview

Show child attributesHide child attributes

enabledbooleanPublic Preview: (Required, Mutable) Whether to enable auto full refresh or not.

min_interval_hoursint32Public Preview: (Optional, Mutable) Specify the minimum interval in hours between the timestamp at which a table was last full refreshed and the current timestamp for triggering auto full If unspecified and autoFullRefresh is enabled then by default min_interval_hours is 24 hours.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview

Catalog-level source configuration parameters

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name

postgresobjectRequiredPublic Preview: Postgres-specific catalog-level configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

Response

pipeline_idstring: The unique identifier for the newly created pipeline. Only returned when dry_run is false.

effective_settingsobject

Only returned when dry_run is true.

Show child attributesHide child attributes

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

spark_env_varsobject

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

s3objectRequired: destination and either region or endpoint should also be provided. e.g. { "s3": { "destination" : "s3://init-scripts/my_script.sh", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobjectRequired

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectRequiredPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectRequiredPublic Preview: Select all tables from a specific source schema.

tableobjectRequiredPublic Preview: Select a specific source table.

reportobjectRequiredPublic Preview: Select a specific source report.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

auto_full_refresh_policyobjectPublic Preview: (Optional, Mutable) Policy for auto full refresh, if enabled pipeline will automatically try to fix issues by doing a full refresh on the table in the retry run. auto_full_refresh_policy in table configuration will override the above level auto_full_refresh_policy. For example, { "auto_full_refresh_policy": { "enabled": true, "min_interval_hours": 23, } } If unspecified, auto full refresh is disabled.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview: Catalog-level source configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

View as Markdown

Request

{
  "clusters": [
    {
      "autoscale": {
        "max_workers": 5,
        "min_workers": 1,
        "mode": "ENHANCED"
      },
      "label": "default"
    }
  ],
  "continuous": false,
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/SDP Notebooks/Spark Declarative Pipelines quickstart (SQL)"
      }
    }
  ],
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data"
}

Response

{
  "pipeline_id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5"
}

Update GA

PUT /api/2.0/pipelines/{pipeline_id}

Updates a pipeline with the supplied configuration.

API scopes: pipelines

Parameters

pipeline_idstringpath: Unique identifier for this pipeline.

Request body

allow_duplicate_namesboolean: If false, deployment will fail if name has changed and conflicts the name of another pipeline.
Default: false

expected_last_modifiedint64: If present, the last-modified time of the pipeline settings before the edit. If the settings were modified after that time, then the request will fail with a conflict.

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

parametersobjectBeta: Key/value map of default parameters to use for pipeline execution. Maximum total size: 10k characters (JSON format)

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

spark_env_varsobject

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: Flag to enable server side encryption, false by default.

encryption_typestring: The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobjectRequired

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectRequiredPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectRequiredPublic Preview

Select all tables from a specific source schema.

Show child attributesHide child attributes

source_catalogstringPublic Preview: The source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store tables.

destination_schemastringPublic Preview: Required. Destination schema to store tables in. Tables with the same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in this schema and override the table_configuration defined in the IngestionPipelineDefinition object.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

fanout_optionsobjectBeta: Fanout options for multi-table routing from streaming sources. When set, records are routed to destination tables based on a per-record routing key. The key value becomes the table name: {destination_catalog}.{destination_schema}.{key_value}.

tableobjectRequiredPublic Preview

Select a specific source table.

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Might be optional depending on the type of source.

source_tablestringPublic Preview: Table name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Optional. Destination table name. The pipeline fails if a table with that name already exists. If not set, the source table name is used.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object and the SchemaSpec.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

reportobjectRequiredPublic Preview

Select a specific source report.

Show child attributesHide child attributes

source_urlstringPublic Preview: Required. Report URL in the source system.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Required. Destination table name. The pipeline fails if a table with that name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

Show child attributesHide child attributes

cursor_columnsarray of stringPublic Preview: The names of the monotonically increasing columns in the source table that are used to enable the table to be read and ingested incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the sequence_by behavior. You can still explicitly set sequence_by to override this default.

deletion_conditionstringPublic Preview: Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to hard_deletion_sync_interval_in_seconds, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table.

hard_deletion_sync_min_interval_in_secondsint64Beta: Specifies the minimum interval (in seconds) between snapshots on primary keys for detecting and synchronizing hard deletions—i.e., rows that have been physically removed from the source table. This interval acts as a lower bound. If ingestion runs less frequently than this value, hard deletion synchronization will align with the actual ingestion frequency instead of happening more often. If not set, hard deletion synchronization via snapshots is disabled. This field is mutable and can be updated without triggering a full snapshot.

auto_full_refresh_policyobjectPublic Preview

Show child attributesHide child attributes

enabledbooleanPublic Preview: (Required, Mutable) Whether to enable auto full refresh or not.

min_interval_hoursint32Public Preview: (Optional, Mutable) Specify the minimum interval in hours between the timestamp at which a table was last full refreshed and the current timestamp for triggering auto full If unspecified and autoFullRefresh is enabled then by default min_interval_hours is 24 hours.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview

Catalog-level source configuration parameters

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name

postgresobjectRequiredPublic Preview: Postgres-specific catalog-level configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

View as Markdown

Request

{
  "clusters": [
    {
      "autoscale": {
        "max_workers": 5,
        "min_workers": 1,
        "mode": "ENHANCED"
      },
      "label": "default"
    }
  ],
  "continuous": false,
  "id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5",
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/SDP Notebooks/Spark Declarative Pipelines quickstart (SQL)"
      }
    }
  ],
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data",
  "target": "wikipedia_quickstart_data"
}

Response

{}

Delete GA

DELETE /api/2.0/pipelines/{pipeline_id}

Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline deletion will cascade to all pipeline tables. Please reach out to <Databricks> support for assistance to undo this action.

API scopes: pipelines

Parameters

pipeline_idstringpath

cascadebooleanquery: If false, pipeline deletion will not cascade to its datasets (MVs, STs, Views). By default, this parameter will be true and all tables will be deleted with the pipeline.

Clone GA

POST /api/2.0/pipelines/{pipeline_id}/clone

Creates a new pipeline using Unity Catalog from a pipeline using Hive Metastore. This method returns the ID of the newly created clone. Additionally, this method starts an update for the newly created pipeline.

API scopes: pipelines

Parameters

pipeline_idstringpath: Source pipeline to clone from

Request body

expected_last_modifiedint64: If present, the last-modified time of the pipeline settings before the clone. If the settings were modified after that time, then the request will fail with a conflict.

allow_duplicate_namesboolean: If false, deployment will fail if name conflicts with that of another pipeline.
Default: false

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring: Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity. See [[AutoAZHelper.scala]] for more details. The list of available zones as well as the default value can be found by using the List Zones_ method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

***internal If this field is ommitted, we will pull in the default from the conf if it exists.

spot_bid_price_percentint32

***internal The default value and documentation here should be kept consistent with CommonConf.defaultSpotBidPricePercent and CommonConf.maxSpotBidPricePercent.

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32

ebs_volume_throughputint32

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

spark_env_varsobject

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired

destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }

Show child attributesHide child attributes

destinationstring: dbfs destination, e.g. dbfs:/my/path

s3objectRequired

Show child attributesHide child attributes

destinationstring: S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring: S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring: S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean: Flag to enable server side encryption, false by default.

encryption_typestring: The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring: Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring: Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring: Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

fileobjectRequired

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes

pathstring: The absolute path of the source code.

globobjectRequiredPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes

includestringPublic Preview

The source code to include for pipelines

Examples:

/Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
/Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
/Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes

schemaobjectRequiredPublic Preview

Select all tables from a specific source schema.

Show child attributesHide child attributes

source_catalogstringPublic Preview: The source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store tables.

destination_schemastringPublic Preview: Required. Destination schema to store tables in. Tables with the same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in this schema and override the table_configuration defined in the IngestionPipelineDefinition object.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

fanout_optionsobjectBeta: Fanout options for multi-table routing from streaming sources. When set, records are routed to destination tables based on a per-record routing key. The key value becomes the table name: {destination_catalog}.{destination_schema}.{key_value}.

tableobjectRequiredPublic Preview

Select a specific source table.

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview: Schema name in the source database. Might be optional depending on the type of source.

source_tablestringPublic Preview: Table name in the source database. Currently required; this field will become optional in an upcoming release, since some source types (for example streaming / message-bus connectors) do not use it. When that change ships, this field's type in the generated SDKs and CLI will change from required to optional (nullable); clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Optional. Destination table name. The pipeline fails if a table with that name already exists. If not set, the source table name is used.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object and the SchemaSpec.

connector_optionsobjectPublic Preview: (Optional) Source Specific Connector Options

reportobjectRequiredPublic Preview

Select a specific source report.

Show child attributesHide child attributes

source_urlstringPublic Preview: Required. Report URL in the source system.

destination_catalogstringPublic Preview: Required. Destination catalog to store table.

destination_schemastringPublic Preview: Required. Destination schema to store table.

destination_tablestringPublic Preview: Required. Destination table name. The pipeline fails if a table with that name already exists.

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings override the table_configuration defined in the IngestionPipelineDefinition object.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

Show child attributesHide child attributes

scd_typestringPublic Preview: Default: SCD_TYPE_1
Values: SCD_TYPE_UNSPECIFIED, SCD_TYPE_1, SCD_TYPE_2, APPEND_ONLY

primary_keysarray of stringPublic Preview: The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview: The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview: A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview: A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview: (Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview

Show child attributesHide child attributes

cursor_columnsarray of stringPublic Preview: The names of the monotonically increasing columns in the source table that are used to enable the table to be read and ingested incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the sequence_by behavior. You can still explicitly set sequence_by to override this default.

deletion_conditionstringPublic Preview: Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to hard_deletion_sync_interval_in_seconds, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table.

hard_deletion_sync_min_interval_in_secondsint64Beta: Specifies the minimum interval (in seconds) between snapshots on primary keys for detecting and synchronizing hard deletions—i.e., rows that have been physically removed from the source table. This interval acts as a lower bound. If ingestion runs less frequently than this value, hard deletion synchronization will align with the actual ingestion frequency instead of happening more often. If not set, hard deletion synchronization via snapshots is disabled. This field is mutable and can be updated without triggering a full snapshot.

auto_full_refresh_policyobjectPublic Preview

Show child attributesHide child attributes

enabledbooleanPublic Preview: (Required, Mutable) Whether to enable auto full refresh or not.

min_interval_hoursint32Public Preview: (Optional, Mutable) Specify the minimum interval in hours between the timestamp at which a table was last full refreshed and the current timestamp for triggering auto full If unspecified and autoFullRefresh is enabled then by default min_interval_hours is 24 hours.

table_propertiesobjectBeta: Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta: Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
Default: false

clustering_columnsarray of stringBeta: List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta: (Optional) Name of the struct column added to each ingested record to hold per row source metadata.

source_configurationsarray of objectPublic Preview

Top-level source configurations

Show child attributesHide child attributes

catalogobjectPublic Preview

Catalog-level source configuration parameters

Show child attributesHide child attributes

source_catalogstringPublic Preview: Source catalog name

postgresobjectRequiredPublic Preview: Postgres-specific catalog-level configuration parameters

full_refresh_windowobjectPublic Preview

(Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

Show child attributesHide child attributes

start_hourint32Public Preview: An integer between 0 and 23 denoting the start hour for the window in the 24-hour day.

days_of_weekarray of stringPublic Preview: Days of week in which the window is allowed to happen If not specified all days of the week will be used.
Values: DAY_OF_WEEK_UNSPECIFIED, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY

time_zone_idstringPublic Preview: Time zone id of window. See https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html for details. If not specified, UTC will be used.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview

Show child attributesHide child attributes

catalog_namestringPublic Preview: (Required, Immutable) The name of the catalog for the connector's staging storage location.

schema_namestringPublic Preview: (Required, Immutable) The name of the schema for the connector's staging storage location.

volume_namestringPublic Preview: (Optional) The Unity Catalog-compatible name for the storage location. This is the volume to use for the data that is extracted by the connector. Spark Declarative Pipelines system will automatically create the volume under the catalog and schema. For Combined Cdc Managed Ingestion pipelines default name for the volume would be : __databricks_ingestion_gateway_staging_data-$pipelineId

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

Show child attributesHide child attributes

quartz_cron_schedulestring

timezone_idstring

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

clone_modestring: The type of clone to perform. Currently, only deep copies are supported
Values: MIGRATE_TO_UC

Response

pipeline_idstring: The pipeline id of the cloned pipeline

View as Markdown

Request

{
  "catalog": "main",
  "clone_mode": "MIGRATE_TO_UC",
  "name": "My Pipeline [UC]",
  "target": "my_target"
}

Response

{
  "pipeline_id": "b23de4f5-1bc2-2bcd-2b3c-2b3cde4f5gh6"
}

Apply Environment Public Preview

POST /api/2.0/pipelines/{pipeline_id}/environment/apply

Applies the current pipeline environment onto the pipeline compute. The environment applied can be used by subsequent dev-mode updates.

API scopes: pipelines

Parameters

pipeline_idstringpath

Events GA

GET /api/2.0/pipelines/{pipeline_id}/events

Retrieves events for a pipeline.

API scopes: pipelines

Parameters

pipeline_idstringpath: The pipeline to return events for.

page_tokenstringquery: Page token returned by previous call. This field is mutually exclusive with all fields in this request except max_results. An error is returned if any fields other than max_results are set when this field is set.

max_resultsint32query: Max number of entries to return in a single page. The system may return fewer than max_results events in a response, even if there are more events available.
Constraints: <= 1000

order_byarray of stringquery: A string indicating a sort order by timestamp for the results, for example, ["timestamp asc"]. The sort order can be ascending or descending. By default, events are returned in descending order by timestamp.

filterstringquery

Criteria to select a subset of results, expressed using a SQL-like syntax. The supported filters are:

level='INFO' (or WARN or ERROR)
level in ('INFO', 'WARN')
id='[event-id]'
timestamp > 'TIMESTAMP' (or >=,<,<=,=)

Composite expressions are supported, for example: level in ('ERROR', 'WARN') AND timestamp> '2021-07-22T06:37:33.083Z'

Response

eventsarray of object

The list of events matching the request criteria.

Show child attributesHide child attributes

idstring: A time-based, globally unique id.

sequenceobject

A sequencing object to identify and order events.

Show child attributesHide child attributes

data_plane_idobject

the ID assigned by the data plane.

Show child attributesHide child attributes

instancestring: The instance name of the data plane emitting an event.

seq_noint64: A sequence number, unique and increasing within the data plane instance.

control_plane_seq_noint64: A sequence number, unique and increasing per pipeline.

originobject

Describes where the event originates from.

Show child attributesHide child attributes

cloudstring: The cloud provider, e.g., AWS or Azure.

regionstring: The cloud region.

org_idint64: The org id of the user. Unique within a cloud.

pipeline_idstring: The id of the pipeline. Globally unique.

pipeline_namestring: The name of the pipeline. Not unique.

cluster_idstring: The id of the cluster where an execution happens. Unique within a region.

update_idstring: The id of an execution. Globally unique.

maintenance_idstring: The id of a maintenance run. Globally unique.

table_idstring: The id of a (delta) table. Globally unique.

dataset_namestring: The name of a dataset. Unique within a pipeline.

flow_idstring: The id of the flow. Globally unique. Incremental queries will generally reuse the same id while complete queries will have a new id per update.

flow_namestring: The name of the flow. Not unique.

batch_idint64: The id of a batch. Unique within a flow.

request_idstring: The id of the request that caused an update.

uc_resource_idstring: The Unity Catalog id of the MV or ST being updated.

hoststring: The optional host name where the event was triggered

materialization_namestring: Materialization name.

ingestion_source_connection_namestringPublic Preview: The name of the source UC connection (if known) from whose data ingestion is described by this event.

ingestion_source_catalog_namestringPublic Preview: The name of the source catalog name (if known) from whose data ingestion is described by this event.

ingestion_source_schema_namestringPublic Preview: The name of the source schema name (if known) from whose data ingestion is described by this event.

ingestion_source_table_namestringPublic Preview: The name of the source table name (if known) from whose data ingestion is described by this event.

ingestion_source_table_versionstringPublic Preview: An optional implementation-defined source table version of a dataset being (re)ingested.

timestampstring: The time of the event.

messagestring: The display message associated with the event.

levelstring: The severity level of the event.
Values: INFO, WARN, ERROR, METRICS

errorobject

Information about an error captured by the event.

Show child attributesHide child attributes

exceptionsarray of object

The exception thrown for this error, with its chain of cause.

Show child attributesHide child attributes

class_namestring: Runtime class of the exception

messagestring: Exception message

stackarray of object: Stack trace consisting of a list of stack frames

fatalboolean: Whether this error is considered fatal, that is, unrecoverable.

event_typestring: The event type. Should always correspond to the details

maturity_levelstring: Maturity level for event_type.
Values: STABLE, EVOLVING, DEPRECATED

truncationobject

Information about which fields were truncated from this event due to size constraints. If empty or absent, no truncation occurred. See https://docs.databricks.com/en/ldp/monitor-event-logs for information on retrieving complete event data.

Show child attributesHide child attributes

truncated_fieldsarray of object

List of fields that were truncated from this event. If empty or absent, no truncation occurred.

Show child attributesHide child attributes

field_namestring: The name of the truncated field (e.g., "error"). Corresponds to field names in PipelineEvent.

next_page_tokenstring: If present, a token to fetch the next page of events.

prev_page_tokenstring: If present, a token to fetch the previous page of events.

View as Markdown

Response

{
  "events": [
    {
      "id": "string",
      "sequence": {},
      "origin": {},
      "timestamp": "string",
      "message": "string",
      "level": "string",
      "error": {},
      "event_type": "string",
      "maturity_level": "string",
      "truncation": {}
    }
  ],
  "next_page_token": "string",
  "prev_page_token": "string"
}

Stop GA

POST /api/2.0/pipelines/{pipeline_id}/stop

Stops the pipeline by canceling the active update. If there is no active update for the pipeline, this request is a no-op.

API scopes: pipelines

Parameters

pipeline_idstringpath

List Updates GA

GET /api/2.0/pipelines/{pipeline_id}/updates

List updates for an active pipeline.

API scopes: pipelines

Parameters

pipeline_idstringpath: The pipeline to return updates for.

page_tokenstringquery: Page token returned by previous call

max_resultsint32query: Max number of entries to return in a single page.

until_update_idstringquery: If present, returns updates until and including this update_id.

Response

updatesarray of object

Show child attributesHide child attributes

pipeline_idstring: The ID of the pipeline.

update_idstring: The ID of this update.

configobject

The pipeline configuration with system defaults applied where unspecified by the user. Not returned by ListUpdates.

Show child attributesHide child attributes

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired: The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

fileobjectRequired: The path to a file that defines a pipeline and is stored in the Databricks Repos.

globobjectRequiredPublic Preview: The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview: Required. Settings specifying tables to replicate and the destination for the replicated tables.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

source_configurationsarray of objectPublic Preview: Top-level source configurations

full_refresh_windowobjectPublic Preview: (Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview: (Optional) Location of staged data storage. This is required for migration from Cdc Managed Ingestion Pipeline with Gateway pipeline to Combined Cdc Managed Ingestion Pipeline. If not specified, the volume for staged data will be created in catalog and schema/target specified in the top level pipeline definition.

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

causestring: What triggered this update.
Values: API_CALL, RETRY_ON_FAILURE, SERVICE_UPGRADE, SCHEMA_CHANGE, JOB_TASK, USER_ACTION, INFRASTRUCTURE_MAINTENANCE

statestring: The update state.
Values: QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, CANCELED

cluster_idstring: The ID of the cluster that the update is running on.

creation_timeint64: The time when this update was created.

full_refreshboolean: If true, this update will reset all tables before running.

refresh_selectionarray of string: A list of tables to update without fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

full_refresh_selectionarray of string: A list of tables to update with fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

validate_onlyboolean: If true, this update only validates the correctness of pipeline source code but does not materialize or publish any datasets.

modestring: Indicates whether the update is either part of a continuous job run, or running in legacy continuous pipeline mode. Returned only for GetUpdate; not populated in ListUpdates responses.
Values: DEFAULT, CONTINUOUS

parametersobjectBeta: Key/value map of parameters used to initiate the update

next_page_tokenstring: If present, then there are more results, and this a token to be used in a subsequent request to fetch the next page.

prev_page_tokenstring: If present, then this token can be used in a subsequent request to fetch the previous page.

View as Markdown

Response

{
  "updates": [
    {
      "pipeline_id": "string",
      "update_id": "string",
      "config": {},
      "cause": "string",
      "state": "string",
      "cluster_id": "string",
      "creation_time": 0,
      "full_refresh": true,
      "refresh_selection": [
        "string"
      ],
      "full_refresh_selection": [
        "string"
      ],
      "validate_only": true,
      "mode": "string",
      "parameters": {}
    }
  ],
  "next_page_token": "string",
  "prev_page_token": "string"
}

Start GA

POST /api/2.0/pipelines/{pipeline_id}/updates

Starts a new update for the pipeline. If there is already an active update for the pipeline, the request will fail and the active update will remain running.

API scopes: pipelines

Parameters

pipeline_idstringpath

Request body

full_refreshboolean: If true, this update will reset all tables before running.

causestring: Values: API_CALL, RETRY_ON_FAILURE, SERVICE_UPGRADE, SCHEMA_CHANGE, JOB_TASK, USER_ACTION, INFRASTRUCTURE_MAINTENANCE

refresh_selectionarray of string: A list of tables to update without fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

full_refresh_selectionarray of string: A list of tables to update with fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

reset_checkpoint_selectionarray of string: A list of flows for which this update should reset the streaming checkpoint. This selection will not clear the data in the flow's target table. Flows in this list may also appear in refresh_selection and full_refresh_selection.

validate_onlyboolean: If true, this update only validates the correctness of pipeline source code but does not materialize or publish any datasets.

parametersobjectBeta: Key/value map of parameters to pass to the pipeline execution

Response

update_idstring

Get Update GA

GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}

Gets an update from an active pipeline.

API scopes: pipelines

Parameters

pipeline_idstringpath: The ID of the pipeline.

update_idstringpath: The ID of the update.

Response

updateobject

The current update info.

Show child attributesHide child attributes

pipeline_idstring: The ID of the pipeline.

update_idstring: The ID of this update.

configobject

The pipeline configuration with system defaults applied where unspecified by the user. Not returned by ListUpdates.

Show child attributesHide child attributes

idstring: Unique identifier for this pipeline.

namestring: Friendly identifier for this pipeline.

storagestring: DBFS root directory for storing checkpoints and tables.

configurationobject: String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes

labelstring: A label for the cluster specification, either default to configure the default cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean: Note: This field won't be persisted. Only API users will check this field.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring: The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes

whlstringRequired: URI of the whl to be installed.

notebookobjectRequired: The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

fileobjectRequired: The path to a file that defines a pipeline and is stored in the Databricks Repos.

globobjectRequiredPublic Preview: The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes

connection_namestringRequiredPublic Preview: The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringRequiredPublic Preview: Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanRequiredPublic Preview: Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.
Default: false

objectsarray of objectPublic Preview: Required. Settings specifying tables to replicate and the destination for the replicated tables.

source_typestringPublic Preview: The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.
Values: INGESTION_SOURCE_TYPE_UNSPECIFIED, MYSQL, POSTGRESQL, SQLSERVER, SALESFORCE, BIGQUERY, NETSUITE, WORKDAY_RAAS, GA4_RAW_DATA, SERVICENOW, MANAGED_POSTGRESQL, ORACLE, TERADATA, SHAREPOINT, DYNAMICS365, JIRA, CONFLUENCE, META_MARKETING, ZENDESK

table_configurationobjectPublic Preview: Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

source_configurationsarray of objectPublic Preview: Top-level source configurations

full_refresh_windowobjectPublic Preview: (Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

connector_typestringPublic Preview: (Optional) Connector Type for sources. Ex: CDC, Query Based.
Values: CONNECTOR_TYPE_UNSPECIFIED, CDC, QUERY_BASED

data_staging_optionsobjectPublic Preview: (Optional) Location of staged data storage. This is required for migration from Cdc Managed Ingestion Pipeline with Gateway pipeline to Combined Cdc Managed Ingestion Pipeline. If not specified, the volume for staged data will be created in catalog and schema/target specified in the top level pipeline definition.

triggerobject

Which pipeline trigger to use. Deprecated: Use continuous instead.

Show child attributesHide child attributes

manualobjectRequired

cronobjectRequired

targetstring: Target schema (database) to add tables in this pipeline to. Exactly one of schema or target must be specified. To publish to Unity Catalog, also specify catalog. This legacy field is deprecated for pipeline creation in favor of the schema field.

schemastring: The default schema (database) where tables are read from or published to.

filtersobject

Filters on which Pipeline packages to include in the deployed graph.

Show child attributesHide child attributes

includearray of string: Paths to include.

excludearray of string: Paths to exclude.

continuousboolean

Whether the pipeline is continuous or triggered. This replaces trigger.

developmentboolean: Whether the pipeline is in Development mode. Defaults to false.

photonboolean: Whether Photon is enabled for this pipeline.

editionstring: Pipeline product edition.

channelstring: SDP Release Channel that specifies which version to use.

catalogstring: A catalog in Unity Catalog to publish data from this pipeline to. If target is specified, tables in this pipeline are published to a target schema inside catalog (for example, catalog.target.table). If target is not specified, no data is published to Unity Catalog.

notificationsarray of object

List of notification settings for this pipeline.

Show child attributesHide child attributes

email_recipientsarray of string: A list of email addresses notified when a configured alert is triggered.

alertsarray of string

A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are:

on-update-success: A pipeline update completes successfully.
on-update-failure: Each time a pipeline update fails.
on-update-fatal-failure: A pipeline update fails with a non-retryable (fatal) error.
on-flow-failure: A single data flow fails.

serverlessboolean: Whether serverless compute is enabled for this pipeline.

deploymentobject

Deployment type of this pipeline.

Show child attributesHide child attributes

kindstring: The deployment method that manages the pipeline.
Values: BUNDLE

metadata_file_pathstring: The path to the file containing metadata about the deployment.

budget_policy_idstringPublic Preview: Budget policy of this pipeline.

tagsobject: A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline.

event_logobject

Event log configuration for this pipeline

Show child attributesHide child attributes

namestring: The name the event log is published to in UC.

schemastring: The UC schema the event log is published under.

catalogstring: The UC catalog the event log is published under.

root_pathstringPublic Preview: Root path for this pipeline. This is used as the root directory when editing the pipeline in the <Databricks> user interface and it is added to sys.path when executing Python sources during pipeline execution.
Example: /Workspace/Users/user.name/my-pipeline

environmentobjectPublic Preview

Environment specification for this pipeline used to install dependencies.

Show child attributesHide child attributes

dependenciesarray of stringPublic Preview: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be <requirement specifier>, <archive url/path>, <local project path>(WSFS or Volumes in <Databricks>), <vcs project url>

environment_versionstringBeta

The value should be a string representing the environment version number, for example: "4".

causestring: What triggered this update.
Values: API_CALL, RETRY_ON_FAILURE, SERVICE_UPGRADE, SCHEMA_CHANGE, JOB_TASK, USER_ACTION, INFRASTRUCTURE_MAINTENANCE

statestring: The update state.
Values: QUEUED, CREATED, WAITING_FOR_RESOURCES, INITIALIZING, RESETTING, SETTING_UP_TABLES, RUNNING, STOPPING, COMPLETED, FAILED, CANCELED

cluster_idstring: The ID of the cluster that the update is running on.

creation_timeint64: The time when this update was created.

full_refreshboolean: If true, this update will reset all tables before running.

refresh_selectionarray of string: A list of tables to update without fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

full_refresh_selectionarray of string: A list of tables to update with fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh.

validate_onlyboolean: If true, this update only validates the correctness of pipeline source code but does not materialize or publish any datasets.

modestring: Indicates whether the update is either part of a continuous job run, or running in legacy continuous pipeline mode. Returned only for GetUpdate; not populated in ListUpdates responses.
Values: DEFAULT, CONTINUOUS

parametersobjectBeta: Key/value map of parameters used to initiate the update

View as Markdown

Response

{
  "update": {
    "cause": "API_CALL",
    "config": {
      "clusters": [
        {
          "autoscale": {
            "max_workers": 5,
            "min_workers": 1,
            "mode": "ENHANCED"
          },
          "label": "default"
        }
      ],
      "configuration": {
        "pipelines.numStreamRetryAttempts": "5"
      },
      "continuous": false,
      "development": false,
      "id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5",
      "libraries": [
        {
          "notebook": {
            "path": "/Users/username/SDP Notebooks/Spark Declarative Pipelines quickstart (SQL)"
          }
        }
      ],
      "name": "Wikipedia pipeline (SQL)",
      "storage": "/Users/username/data",
      "target": "wikipedia_quickstart_data"
    },
    "creation_time": 1628815050279,
    "full_refresh": true,
    "pipeline_id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5",
    "request_id": "a83d9f7c-d798-4fd5-aa39-301b6e6f4429",
    "state": "COMPLETED",
    "update_id": "9a84f906-fc51-11eb-9a03-0242ac130003"
  }
}

PipelineSpec object​

Get GA​

Parameters​

Response​

List GA​

Parameters​

Response​

Create GA​

Request body​

Response​

Update GA​

Parameters​

Request body​

Delete GA​

Parameters​

Clone GA​

Parameters​

Request body​

Response​

Apply Environment Public Preview​

Parameters​

Events GA​

Parameters​

Response​

Stop GA​

Parameters​

List Updates GA​

Parameters​

Response​

Start GA​

Parameters​

Request body​

Response​

Get Update GA​

Parameters​

Response​

PipelineSpec object

Get GA

Parameters

Response

List GA

Parameters

Response

Create GA

Request body

Response

Update GA

Parameters

Request body

Delete GA

Parameters

Clone GA

Parameters

Request body

Response

Apply Environment Public Preview

Parameters

Events GA

Parameters

Response

Stop GA

Parameters

List Updates GA

Parameters

Response

Start GA

Parameters

Request body

Response

Get Update GA

Parameters

Response