# Get Update Launch stage: GA `GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}` Gets an update from an active pipeline. API scopes: pipelines ## Path parameters - `pipeline_id` (string, optional) The ID of the pipeline. - `update_id` (string, optional) The ID of the update. ## Returns - `update` (object, optional) The current update info. - `pipeline_id` (string, optional) The ID of the pipeline. - `update_id` (string, optional) The ID of this update. - `config` (object, optional) The pipeline configuration with system defaults applied where unspecified by the user. Not returned by ListUpdates. - `id` (string, optional) Unique identifier for this pipeline. - `name` (string, optional) Friendly identifier for this pipeline. - `storage` (string, optional) DBFS root directory for storing checkpoints and tables. - `configuration` (object, optional) String-String configuration for this pipeline execution. - `clusters` (array of object, optional) Cluster settings for this pipeline deployment. - `label` (string, optional) A label for the cluster specification, either `default` to configure the default cluster, or `maintenance` to configure the maintenance cluster. This field is optional. The default value is `default`. - `apply_policy_default_values` (boolean, optional) Note: This field won't be persisted. Only API users will check this field. - `spark_conf` (object, optional) An object containing a set of optional, user-specified Spark configuration key-value pairs. See :method:clusters/create for more details. - `aws_attributes` (object, optional) Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used. - `azure_attributes` (object, optional) Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used. - `gcp_attributes` (object, optional) Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used. - `node_type_id` (string, optional) This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the :method:clusters/listNodeTypes API call. - `driver_node_type_id` (string, optional) The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as `node_type_id` defined above. - `ssh_public_keys` (array of string, optional) SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name `ubuntu` on port `2200`. Up to 10 keys can be specified. - `custom_tags` (object, optional) Additional tags for cluster resources. will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to `default_tags`. Notes: - Currently, allows at most 45 custom tags - Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags - `cluster_log_conf` (object, optional) The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every `5 mins`. The destination of driver logs is `$destination/$clusterId/driver`, while the destination of executor logs is `$destination/$clusterId/executor`. - `spark_env_vars` (object, optional) An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., `export X='Y'`) while launching the driver and workers. In order to specify an additional set of `SPARK_DAEMON_JAVA_OPTS`, we recommend appending them to `$SPARK_DAEMON_JAVA_OPTS` as shown in the example below. This ensures that all default databricks managed environmental variables are included as well. Example Spark environment variables: `{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}` or `{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}` - `init_scripts` (array of object, optional) The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If `cluster_log_conf` is specified, init script logs are sent to `//init_scripts`. - `instance_pool_id` (string, optional) The optional ID of the instance pool to which the cluster belongs. - `policy_id` (string, optional) The ID of the cluster policy used to create the cluster if applicable. - `enable_local_disk_encryption` (boolean, optional) Whether to enable local disk encryption for the cluster. - `driver_instance_pool_id` (string, optional) The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned. - `num_workers` (int32, required) Number of worker nodes that this cluster should have. A cluster has one Spark Driver and `num_workers` Executors for a total of `num_workers` + 1 Spark nodes. Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in `spark_info` will gradually increase from 5 to 10 as the new nodes are provisioned. - `autoscale` (object, required) Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later. - `libraries` (array of object, optional) Libraries or code needed by this deployment. - `whl` (string, required) URI of the whl to be installed. - `notebook` (object, required) The path to a notebook that defines a pipeline and is stored in the workspace. - `file` (object, required) The path to a file that defines a pipeline and is stored in the Databricks Repos. - `glob` (object, required, Public Preview) The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends `/**`. This field cannot be used together with `notebook` or `file`. - `ingestion_definition` (object, optional, Public Preview) The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings. - `connection_name` (string, required, Public Preview) The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline. - `ingestion_gateway_id` (string, required, Public Preview) Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline. - `ingest_from_uc_foreign_catalog` (boolean, required, Public Preview) Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The `source_catalog` fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from. Default: `false` - `objects` (array of object, optional, Public Preview) Required. Settings specifying tables to replicate and the destination for the replicated tables. - `source_type` (string, optional, Public Preview) The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided. Possible values: `INGESTION_SOURCE_TYPE_UNSPECIFIED`, `MYSQL`, `POSTGRESQL`, `SQLSERVER`, `SALESFORCE`, `BIGQUERY`, `NETSUITE`, `WORKDAY_RAAS`, `GA4_RAW_DATA`, `SERVICENOW`, `MANAGED_POSTGRESQL`, `ORACLE`, `TERADATA`, `SHAREPOINT`, `DYNAMICS365`, `JIRA`, `CONFLUENCE`, `META_MARKETING`, `ZENDESK` - `table_configuration` (object, optional, Public Preview) Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline. - `source_configurations` (array of object, optional, Public Preview) Top-level source configurations - `full_refresh_window` (object, optional, Public Preview) (Optional) A window that specifies a set of time ranges for snapshot queries in CDC. - `connector_type` (string, optional, Public Preview) (Optional) Connector Type for sources. Ex: CDC, Query Based. Possible values: `CONNECTOR_TYPE_UNSPECIFIED`, `CDC`, `QUERY_BASED` - `data_staging_options` (object, optional, Public Preview) (Optional) Location of staged data storage. This is required for migration from Cdc Managed Ingestion Pipeline with Gateway pipeline to Combined Cdc Managed Ingestion Pipeline. If not specified, the volume for staged data will be created in catalog and schema/target specified in the top level pipeline definition. - `trigger` (object, optional) Which pipeline trigger to use. Deprecated: Use `continuous` instead. - `manual` (object, required) - `cron` (object, required) - `target` (string, optional) Target schema (database) to add tables in this pipeline to. Exactly one of `schema` or `target` must be specified. To publish to Unity Catalog, also specify `catalog`. This legacy field is deprecated for pipeline creation in favor of the `schema` field. - `schema` (string, optional) The default schema (database) where tables are read from or published to. - `filters` (object, optional) Filters on which Pipeline packages to include in the deployed graph. - `include` (array of string, optional) Paths to include. - `exclude` (array of string, optional) Paths to exclude. - `continuous` (boolean, optional) Whether the pipeline is continuous or triggered. This replaces `trigger`. Deprecated: wrap the pipeline in a continuous job instead, which also lets you take advantage of job-level settings such as performance mode. When the pipeline is started by a continuous job, the job's setting takes precedence and this field is ignored. - `development` (boolean, optional) Whether the pipeline is in Development mode. Defaults to false. - `photon` (boolean, optional) Whether Photon is enabled for this pipeline. - `edition` (string, optional) Pipeline product edition. - `channel` (string, optional) SDP Release Channel that specifies which version to use. - `catalog` (string, optional) A catalog in Unity Catalog to publish data from this pipeline to. If `target` is specified, tables in this pipeline are published to a `target` schema inside `catalog` (for example, `catalog`.`target`.`table`). If `target` is not specified, no data is published to Unity Catalog. - `notifications` (array of object, optional) List of notification settings for this pipeline. - `email_recipients` (array of string, optional) A list of email addresses notified when a configured alert is triggered. - `alerts` (array of string, optional) A list of alerts that trigger the sending of notifications to the configured destinations. The supported alerts are: * `on-update-success`: A pipeline update completes successfully. * `on-update-failure`: Each time a pipeline update fails. * `on-update-fatal-failure`: A pipeline update fails with a non-retryable (fatal) error. * `on-flow-failure`: A single data flow fails. - `serverless` (boolean, optional) Whether serverless compute is enabled for this pipeline. - `deployment` (object, optional) Deployment type of this pipeline. - `kind` (string, optional) The deployment method that manages the pipeline. Possible values: `BUNDLE` - `metadata_file_path` (string, optional) The path to the file containing metadata about the deployment. - `budget_policy_id` (string, optional, Public Preview) Budget policy of this pipeline. - `tags` (object, optional) A map of tags associated with the pipeline. These are forwarded to the cluster as cluster tags, and are therefore subject to the same limitations. A maximum of 25 tags can be added to the pipeline. - `event_log` (object, optional) Event log configuration for this pipeline - `name` (string, optional) The name the event log is published to in UC. - `schema` (string, optional) The UC schema the event log is published under. - `catalog` (string, optional) The UC catalog the event log is published under. - `root_path` (string, optional, Public Preview) Root path for this pipeline. This is used as the root directory when editing the pipeline in the user interface and it is added to sys.path when executing Python sources during pipeline execution. Example: `/Workspace/Users/user.name/my-pipeline` - `environment` (object, optional, Public Preview) Environment specification for this pipeline used to install dependencies. - `dependencies` (array of string, optional, Public Preview) List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a pip requirement file line https://pip.pypa.io/en/stable/reference/requirements-file-format/ Allowed dependency could be , , (WSFS or Volumes in ), - `environment_version` (string, optional, Beta) The environment version of the serverless Python environment used to execute customer Python code. Each environment version includes a specific Python version and a curated set of pre-installed libraries with defined versions, providing a stable and reproducible execution environment. supports a three-year lifecycle for each environment version. For available versions and their included packages, see https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ The value should be a string representing the environment version number, for example: `"4"`. - `cause` (string, optional) What triggered this update. Possible values: `API_CALL`, `RETRY_ON_FAILURE`, `SERVICE_UPGRADE`, `SCHEMA_CHANGE`, `JOB_TASK`, `USER_ACTION`, `INFRASTRUCTURE_MAINTENANCE` - `state` (string, optional) The update state. Possible values: `QUEUED`, `CREATED`, `WAITING_FOR_RESOURCES`, `INITIALIZING`, `RESETTING`, `SETTING_UP_TABLES`, `RUNNING`, `STOPPING`, `COMPLETED`, `FAILED`, `CANCELED` - `cluster_id` (string, optional) The ID of the cluster that the update is running on. - `creation_time` (int64, optional) The time when this update was created. - `full_refresh` (boolean, optional) If true, this update will reset all tables before running. - `refresh_selection` (array of string, optional) A list of tables to update without fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh. - `full_refresh_selection` (array of string, optional) A list of tables to update with fullRefresh. If both refresh_selection and full_refresh_selection are empty, this is a full graph update. Full Refresh on a table means that the states of the table will be reset before the refresh. - `validate_only` (boolean, optional) If true, this update only validates the correctness of pipeline source code but does not materialize or publish any datasets. - `mode` (string, optional) Indicates whether the update is either part of a continuous job run, or running in legacy continuous pipeline mode. Returned only for GetUpdate; not populated in ListUpdates responses. Possible values: `DEFAULT`, `CONTINUOUS` - `parameters` (object, optional, Beta) Key/value map of parameters used to initiate the update ## Response ```json { "update": { "cause": "API_CALL", "config": { "clusters": [ { "autoscale": { "max_workers": 5, "min_workers": 1, "mode": "ENHANCED" }, "label": "default" } ], "configuration": { "pipelines.numStreamRetryAttempts": "5" }, "continuous": false, "development": false, "id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5", "libraries": [ { "notebook": { "path": "/Users/username/SDP Notebooks/Spark Declarative Pipelines quickstart (SQL)" } } ], "name": "Wikipedia pipeline (SQL)", "storage": "/Users/username/data", "target": "wikipedia_quickstart_data" }, "creation_time": 1628815050279, "full_refresh": true, "pipeline_id": "a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5", "request_id": "a83d9f7c-d798-4fd5-aa39-301b6e6f4429", "state": "COMPLETED", "update_id": "9a84f906-fc51-11eb-9a03-0242ac130003" } } ```