Run
Run object
- job_idint64
The canonical identifier of the job that contains this run.
- run_idint64
The canonical identifier of the run. This ID is unique across all runs of all jobs.
- creator_user_namestring
The creator user name. This field won’t be included in the response if the user has already been deleted.
- number_in_jobint64
A unique identifier for this job run. This is set to the same value as
run_id.
- original_attempt_run_idint64
If this run is a retry of a prior run attempt, this field contains the run_id of the original attempt; otherwise, it is the same as the run_id.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- scheduleobject
The cron schedule that triggered this run if it was triggered by the periodic scheduler.
Show child attributesHide child attributes
- quartz_cron_expressionstring
A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
- timezone_idstring
A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
- pause_statusstring
Indicate whether this schedule is paused or not.
- cluster_specobject
A snapshot of the job’s cluster specification when this run was created.
Show child attributesHide child attributes
- existing_cluster_idstring
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobject
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstringGA
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectGA
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobject
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobject
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobject
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobject
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobject
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobject
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobject
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- job_cluster_keystring
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstring
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstring
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobject
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }Show child attributesHide child attributes
- packagestring
The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".
- repostring
The repository where the package can be found. If not specified, the default pip index is used.
- mavenobject
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }Show child attributesHide child attributes
- coordinatesstring
Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
- repostring
Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.
- exclusionsarray of string
List of dependences to exclude. For example:
["slf4j:slf4j", "*:hadoop-client"].Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
- cranobject
Specification of a CRAN library to be installed as part of the library
Show child attributesHide child attributes
- packagestring
The name of the CRAN package to install.
- repostring
The repository where the package can be found. If not specified, the default CRAN repo is used.
- whlstring
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstring
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- job_parametersarray of object
Job-level parameters used in the run
Show child attributesHide child attributes
- namestring
The name of the parameter
- defaultstring
The optional default value of the parameter
- valuestring
The value used in the run
- overriding_parametersobject
The parameters used for this run.
Show child attributesHide child attributes
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- triggerstring
- trigger_infoobject
Show child attributesHide child attributes
- run_idint64
The run id of the Run Job task run
- run_namestring
An optional name for the run. The maximum length is 4096 bytes in UTF-8 encoding.
- run_page_urlstring
The URL to the detail page of the run.
- run_typestring
- tasksarray of object
The list of tasks performed by the run. Each task has its own
run_idwhich you can use to callJobsGetOutputto retrieve the run results. If more than 100 tasks are available, you can paginate through them usingjobs/getrun. Use thenext_page_tokenfield at the object root to determine if more results are available.Show child attributesHide child attributes
- run_idint64
The ID of the task run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- run_page_urlstring
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstring
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstring
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstring
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobjectGA
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- resolved_valuesobject
Parameter values including resolved references
Show child attributesHide child attributes
- notebook_taskobject
Show child attributesHide child attributes
- base_parametersobject
- spark_jar_taskobject
Show child attributesHide child attributes
- parametersarray of string
- spark_python_taskobject
- spark_submit_taskobject
- python_wheel_taskobject
Show child attributesHide child attributes
- parametersarray of string
- named_parametersobject
- dbt_taskobject
Show child attributesHide child attributes
- commandsarray of string
- sql_taskobject
Show child attributesHide child attributes
- parametersobject
- run_job_taskobject
Show child attributesHide child attributes
- parametersobject
- job_parametersobject
- condition_taskobject
Show child attributesHide child attributes
- leftstring
- rightstring
- simulation_taskobject
Show child attributesHide child attributes
- parametersobject
- pipeline_taskobjectBeta
Show child attributesHide child attributes
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- descriptionstring
An optional description for this task.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. The key is
task_key, and the value is the name assigned to the dependent task.Show child attributesHide child attributes
- task_keystring
The name of the task this task depends on.
- outcomestring
Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.
- run_ifstring
An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, defaults to
ALL_SUCCESS. Seejobs/createfor a list of possible values.
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- email_notificationsobject
An optional set of email addresses notified when the task run begins or completes. The default behavior is to not send any emails.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
Show child attributesHide child attributes
- metricstring
- opstring
- valueint64
Specifies the threshold value that the health metric should obey to satisfy the health rule.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes. The default behavior is to not send any system notifications. Task webhooks respect the task notification settings.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.Show child attributesHide child attributes
- idstring
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.Show child attributesHide child attributes
- idstring
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.Show child attributesHide child attributes
- idstring
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.Show child attributesHide child attributes
- idstring
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.Show child attributesHide child attributes
- idstring
- environment_keystring
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledbooleanGA
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
Show child attributesHide child attributes
- hardware_acceleratorstringBeta
Hardware accelerator configuration for Serverless GPU workloads.
- notebook_taskobject
The task runs a notebook when the
notebook_taskfield is present.Show child attributesHide child attributes
- notebook_pathstring
The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
- base_parametersobject
Base parameters to be used for each run of this job. If the run is initiated by a call to
jobs/runNow with parameters specified, the two parameters maps are merged. If the same key is specified inbase_parametersand inrun-now, the value fromrun-nowis used. Use Task parameter variables to set parameters containing information about job runs.If the notebook takes a parameter that is not specified in the job’s
base_parametersor therun-nowoverride parameters, the default value from the notebook is used.Retrieve these parameters in a notebook using dbutils.widgets.get.
The JSON representation of this field cannot exceed 1MB.
- sourcestring
Optional location type of the notebook. When set to
WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set toGIT, the notebook will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Notebook is located in <Databricks> workspace.GIT: Notebook is located in cloud Git provider.
- warehouse_idstring
Optional
warehouse_idto run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.
- spark_jar_taskobject
The task runs a JAR when the
spark_jar_taskfield is present.Show child attributesHide child attributes
- jar_uristring
Deprecated since 04/2016. For classic compute, provide a
jarthrough thelibrariesfield instead. For serverless compute, provide ajarthough thejava_dependenciesfield inside theenvironmentslist.See the examples of classic and serverless compute usage at the top of the page.
- main_class_namestring
The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.
The code must use
SparkContext.getOrCreateto obtain a Spark context; otherwise, runs of the job fail.
- parametersarray of string
Parameters passed to the main method.
Use Task parameter variables to set parameters containing information about job runs.
- run_as_replboolean
Deprecated. A value of
falseis no longer supported.
- spark_python_taskobject
The task runs a Python file when the
spark_python_taskfield is present.Show child attributesHide child attributes
- python_filestring
The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with
/. For files stored in a remote repository, the path must be relative. This field is required.
- parametersarray of string
Command line parameters passed to the Python file.
Use Task parameter variables to set parameters containing information about job runs.
- sourcestring
Optional location type of the Python file. When set to
WORKSPACEor not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if thepython_filehas a URI format). When set toGIT, the Python file will be retrieved from a Git repository defined ingit_source.WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.GIT: The Python file is located in a remote Git repository.
- spark_submit_taskobject
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
Show child attributesHide child attributes
- parametersarray of string
Command-line parameters passed to spark submit.
Use Task parameter variables to set parameters containing information about job runs.
- pipeline_taskobject
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.Show child attributesHide child attributes
- pipeline_idstring
The full name of the pipeline task to execute.
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- python_wheel_taskobject
The task runs a Python wheel when the
python_wheel_taskfield is present.Show child attributesHide child attributes
- package_namestring
Name of the package to execute
- entry_pointstring
Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using
$packageName.$entryPoint()
- parametersarray of string
Command-line parameters passed to Python wheel task. Leave it empty if
named_parametersis not null.
- named_parametersobject
Command-line parameters passed to Python wheel task in the form of
["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty ifparametersis not null.
- dbt_taskobject
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.Show child attributesHide child attributes
- project_directorystring
Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.
- commandsarray of string
A list of dbt commands to execute. All commands must start with
dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.
- schemastring
Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the
defaultschema is used.
- warehouse_idstring
ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the
--profiles-dircommand line argument.
- profiles_directorystring
Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.
- catalogstring
Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
- sourcestring
Optional location type of the project directory. When set to
WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set toGIT, the project will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Project is located in <Databricks> workspace.GIT: Project is located in cloud Git provider.
- sql_taskobject
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.Show child attributesHide child attributes
- parametersobject
Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.
- queryobject
If query, indicates that this job must execute a SQL query.
Show child attributesHide child attributes
- query_idstring
The canonical identifier of the SQL query.
- dashboardobject
If dashboard, indicates that this job must refresh a SQL dashboard.
Show child attributesHide child attributes
- dashboard_idstring
The canonical identifier of the SQL dashboard.
- subscriptionsarray of object
If specified, dashboard snapshots are sent to subscriptions.
- custom_subjectstring
Subject of the email sent to subscribers of this task.
- pause_subscriptionsboolean
If true, the dashboard snapshot is not taken, and emails are not sent to subscribers.
- alertobject
If alert, indicates that this job must refresh a SQL alert.
Show child attributesHide child attributes
- alert_idstring
The canonical identifier of the SQL alert.
- subscriptionsarray of object
If specified, alert notifications are sent to subscribers.
- pause_subscriptionsboolean
If true, the alert notifications are not sent to subscribers.
- fileobject
If file, indicates that this job runs a SQL file in a remote Git repository.
Show child attributesHide child attributes
- pathstring
Path of the SQL file. Must be relative if the source is a remote Git repository and absolute for workspace paths.
- sourcestring
Optional location type of the SQL file. When set to
WORKSPACE, the SQL file will be retrieved from the local <Databricks> workspace. When set toGIT, the SQL file will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: SQL file is located in <Databricks> workspace.GIT: SQL file is located in cloud Git provider.
- warehouse_idstring
The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.
- run_job_taskobject
The task triggers another job when the
run_job_taskfield is present.Show child attributesHide child attributes
- job_idint64
ID of the job to trigger.
- job_parametersobject
Job-level parameters used to trigger the job.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- condition_taskobject
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.Show child attributesHide child attributes
- opstring
EQUAL_TO,NOT_EQUALoperators perform string comparison of their operands. This means that“12.0” == “12”will evaluate tofalse.GREATER_THAN,GREATER_THAN_OR_EQUAL,LESS_THAN,LESS_THAN_OR_EQUALoperators perform numeric comparison of their operands.“12.0” >= “12”will evaluate totrue,“10.0” >= “12”will evaluate tofalse.
The boolean comparison to task values can be implemented with operators
EQUAL_TO,NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to“true”or“false”for the comparison.
- leftstring
The left operand of the condition task. Can be either a string value or a job state or parameter reference.
- rightstring
The right operand of the condition task. Can be either a string value or a job state or parameter reference.
- outcomestring
The condition expression evaluation result. Filled in if the task was successfully completed. Can be
"true"or"false"
- for_each_taskobject
The task executes a nested task for every input provided when the
for_each_taskfield is present.Show child attributesHide child attributes
- inputsstring
Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
- concurrencyint32
An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
- taskobject
Configuration for the task that will be run for each element in the array
Show child attributesHide child attributes
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the
run_ifcondition is true. The key istask_key, and the value is the name assigned to the dependent task.
- run_ifstring
An optional value specifying the condition determining whether the task is run once its dependencies have been completed.
ALL_SUCCESS: All dependencies have executed and succeededAT_LEAST_ONE_SUCCESS: At least one dependency has succeededNONE_FAILED: None of the dependencies have failed and at least one was executedALL_DONE: All dependencies have been completedAT_LEAST_ONE_FAILED: At least one dependency failedALL_FAILED: ALl dependencies have failed
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- healthobject
- email_notificationsobject
An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task.
- webhook_notificationsobject
A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.
- descriptionstring
An optional description for this task.
- environment_keystring
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledbooleanGA
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
- notebook_taskobject
The task runs a notebook when the
notebook_taskfield is present.
- spark_jar_taskobject
The task runs a JAR when the
spark_jar_taskfield is present.
- spark_python_taskobject
The task runs a Python file when the
spark_python_taskfield is present.
- spark_submit_taskobject
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
- pipeline_taskobject
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.
- python_wheel_taskobject
The task runs a Python wheel when the
python_wheel_taskfield is present.
- dbt_taskobject
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.
- sql_taskobject
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.
- run_job_taskobject
The task triggers another job when the
run_job_taskfield is present.
- condition_taskobject
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.
- for_each_taskobject
The task executes a nested task for every input provided when the
for_each_taskfield is present.
- clean_rooms_notebook_taskobjectGA
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.
- alert_taskobjectPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.
- power_bi_taskobjectPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.
- dashboard_taskobject
The task refreshes a dashboard and sends a snapshot to subscribers.
- ai_runtime_taskobjectPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
- existing_cluster_idstring
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobject
If new_cluster, a description of a new cluster that is created for each run.
- job_cluster_keystring
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- clean_rooms_notebook_taskobjectGA
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.Show child attributesHide child attributes
- clean_room_namestring
The clean room that the notebook belongs to.
- notebook_namestring
Name of the notebook being run.
- etagstring
Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the
cleanroomassets/getAPI.
- notebook_base_parametersobject
Base parameters to be used for the clean room notebook job.
- alert_taskobjectPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.Show child attributesHide child attributes
- alert_idstringPublic Preview
The alert_id is the canonical identifier of the alert.
- warehouse_idstringPublic Preview
The warehouse_id identifies the warehouse settings used by the alert task.
- workspace_pathstringPublic Preview
The workspace_path is the path to the alert file in the workspace. The path:
- must start with "/Workspace"
- must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.
- subscribersarray of objectPublic Preview
The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.
Show child attributesHide child attributes
- user_namestringPublic Preview
A valid workspace email address.
- destination_idstringPublic Preview
- power_bi_taskobjectPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.Show child attributesHide child attributes
- tablesarray of objectPublic Preview
The tables to be exported to Power BI
Show child attributesHide child attributes
- namestringPublic Preview
The table name in <Databricks>
- catalogstringPublic Preview
The catalog name in <Databricks>
- schemastringPublic Preview
The schema name in <Databricks>
- storage_modestringPublic Preview
The Power BI storage mode of the table
- warehouse_idstringPublic Preview
The SQL warehouse ID to use as the Power BI data source
- power_bi_modelobjectPublic Preview
The semantic model to update
Show child attributesHide child attributes
- workspace_namestringPublic Preview
The name of the Power BI workspace of the model
- model_namestringPublic Preview
The name of the Power BI model
- storage_modestringPublic Preview
The default storage mode of the Power BI model
- authentication_methodstringPublic Preview
How the published Power BI model authenticates to <Databricks>
- overwrite_existingbooleanPublic Preview
Whether to overwrite existing Power BI models
- connection_resource_namestringPublic Preview
The resource name of the UC connection to authenticate from <Databricks> to Power BI
- refresh_after_updatebooleanPublic Preview
Whether the model should be refreshed after the update
- dashboard_taskobject
The task refreshes a dashboard and sends a snapshot to subscribers.
Show child attributesHide child attributes
- subscriptionobject
Optional: subscription configuration for sending the dashboard snapshot.
Show child attributesHide child attributes
- subscribersarray of object
The list of subscribers to send the snapshot of the dashboard to.
- pausedboolean
When true, the subscription will not send emails.
- custom_subjectstring
Optional: Allows users to specify a custom subject line on the email sent to subscribers.
- warehouse_idstring
Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
- dashboard_idstring
The identifier of the dashboard to refresh.
- ai_runtime_taskobjectPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
Show child attributesHide child attributes
- experimentstringPublic Preview
MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set
mlflow_experiment_directory.
- deploymentsarray of objectPublic Preview
Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.
Show child attributesHide child attributes
- command_pathstringPublic Preview
Workspace path of the script to run on each node in this deployment. Upload the script to this path and supply the path here. When the task runs, the file at this path is run on each node; if it fails, the task fails with its exit code.
Example script contents:
Plain Python:
python train.py --epochs 10
Multi-GPU via accelerate:
accelerate launch train.py --config config.yaml
Distributed via torchrun:
torchrun --nproc_per_node=8 train.py
- computeobjectPublic Preview
Compute resources allocated to each node in this deployment.
- namestringPublic Preview
Optional human-readable name for this deployment (for example,
driver,worker,param_server). Used for log and UI display. Distinct names are recommended so deployments can be told apart, but uniqueness is not enforced.
- mlflow_runstringPublic Preview
Optional display name for the MLflow run created under
experiment. If omitted, MLflow generates a default name.
- mlflow_experiment_directorystringPublic Preview
Optional workspace directory under which the MLflow experiment named in
experimentis created. Must start with/Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
- docker_image_urlstringBeta
Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format:
{organization}/{repository}:{tag}
- existing_cluster_idstring
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobject
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstringGA
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectGA
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobject
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobject
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobject
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobject
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobject
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobject
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobject
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- job_cluster_keystring
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstring
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstring
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobject
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }Show child attributesHide child attributes
- packagestring
The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".
- repostring
The repository where the package can be found. If not specified, the default pip index is used.
- mavenobject
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }Show child attributesHide child attributes
- coordinatesstring
Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
- repostring
Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.
- exclusionsarray of string
List of dependences to exclude. For example:
["slf4j:slf4j", "*:hadoop-client"].Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
- cranobject
Specification of a CRAN library to be installed as part of the library
Show child attributesHide child attributes
- packagestring
The name of the CRAN package to install.
- repostring
The repository where the package can be found. If not specified, the default CRAN repo is used.
- whlstring
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstring
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- descriptionstring
Description of the run
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- job_clustersarray of object
A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings. If more than 100 job clusters are available, you can paginate through them using
jobs/getrun.Show child attributesHide child attributes
- job_cluster_keystring
A unique name for the job cluster. This field is required and must be unique within the job.
JobTaskSettingsmay refer to this field to determine which cluster to launch for the task execution.
- new_clusterobject
If new_cluster, a description of a cluster that is created for each task.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstringGA
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectGA
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobject
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3object
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobject
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobject
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobject
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobject
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobject
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobject
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobject
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.
If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task.Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,
git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstring
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstring
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstring
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobjectGA
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- repair_historyarray of object
The repair history of the run.
Show child attributesHide child attributes
- typestring
The repair history item type. Indicates whether a run is the original run or a repair run.
- start_timeint64
The start time of the (repaired) run.
- end_timeint64
The end time of the (repaired) run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- idint64
The ID of the repair. Only returned for the items that represent a repair in
repair_history.
- task_run_idsarray of int64
The run IDs of the task runs that ran as part of this repair history item.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- job_run_idint64
ID of the job run that this run belongs to. For legacy and single-task job runs the field is populated with the job run ID. For task runs, the field is populated with the ID of the job run that the task run belongs to.
- has_moreboolean
Indicates if the run has more array properties (
tasks,job_clusters) that are not shown. They can be accessed viajobs/getrunendpoint. It is only relevant for API 2.2jobs/listrunsrequests withexpand_tasks=true.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
Get GA
GET
Retrieves the metadata of a run.
Large arrays in the results will be paginated when they exceed 100 elements.
A request for a single run will return all properties for that run, and the first 100 elements of array properties (tasks, job_clusters, job_parameters and repair_history).
Use the next_page_token field to check for more results and pass its value as the page_token in subsequent requests.
If any array properties have more than 100 elements, additional results will be returned on subsequent requests. Arrays without additional results will be empty on later pages.
API scopes: jobs
Parameters
- run_idint64query
The canonical identifier of the run for which to retrieve the metadata. This field is required.
- include_historybooleanquery
Whether to include the repair history in the response.
- include_resolved_valuesbooleanquery
Whether to include resolved parameter values in the response.
- page_tokenstringquery
Use
next_page_tokenreturned from the previous GetRun response to request the next page of the run's array properties.
Response
- next_page_tokenstring
A token that can be used to list the next page of array properties.
- job_idint64
The canonical identifier of the job that contains this run.
- run_idint64
The canonical identifier of the run. This ID is unique across all runs of all jobs.
- creator_user_namestring
The creator user name. This field won’t be included in the response if the user has already been deleted.
- number_in_jobint64
A unique identifier for this job run. This is set to the same value as
run_id.
- original_attempt_run_idint64
If this run is a retry of a prior run attempt, this field contains the run_id of the original attempt; otherwise, it is the same as the run_id.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- scheduleobject
The cron schedule that triggered this run if it was triggered by the periodic scheduler.
Show child attributesHide child attributes
- quartz_cron_expressionstring
A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
- timezone_idstring
A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
- pause_statusstring
Indicate whether this schedule is paused or not.
- cluster_specobject
A snapshot of the job’s cluster specification when this run was created.
Show child attributesHide child attributes
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstring
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectRequired
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobjectRequired
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobjectRequired
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobjectRequired
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobjectRequired
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobjectRequired
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobjectRequired
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }Show child attributesHide child attributes
- packagestring
The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".
- repostring
The repository where the package can be found. If not specified, the default pip index is used.
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }Show child attributesHide child attributes
- coordinatesstring
Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
- repostring
Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.
- exclusionsarray of string
List of dependences to exclude. For example:
["slf4j:slf4j", "*:hadoop-client"].Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
Show child attributesHide child attributes
- packagestring
The name of the CRAN package to install.
- repostring
The repository where the package can be found. If not specified, the default CRAN repo is used.
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- job_parametersarray of object
Job-level parameters used in the run
Show child attributesHide child attributes
- namestring
The name of the parameter
- defaultstring
The optional default value of the parameter
- valuestring
The value used in the run
- overriding_parametersobject
The parameters used for this run.
Show child attributesHide child attributes
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- triggerstring
- trigger_infoobject
Show child attributesHide child attributes
- run_idint64
The run id of the Run Job task run
- run_namestring
An optional name for the run. The maximum length is 4096 bytes in UTF-8 encoding.
- run_page_urlstring
The URL to the detail page of the run.
- run_typestring
- tasksarray of object
The list of tasks performed by the run. Each task has its own
run_idwhich you can use to callJobsGetOutputto retrieve the run results. If more than 100 tasks are available, you can paginate through them usingjobs/getrun. Use thenext_page_tokenfield at the object root to determine if more results are available.Show child attributesHide child attributes
- run_idint64
The ID of the task run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- run_page_urlstring
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobject
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- resolved_valuesobject
Parameter values including resolved references
Show child attributesHide child attributes
- notebook_taskobjectRequired
Show child attributesHide child attributes
- base_parametersobject
- spark_jar_taskobjectRequired
Show child attributesHide child attributes
- parametersarray of string
- spark_python_taskobjectRequired
- spark_submit_taskobjectRequired
- python_wheel_taskobjectRequired
Show child attributesHide child attributes
- parametersarray of string
- named_parametersobject
- dbt_taskobjectRequired
Show child attributesHide child attributes
- commandsarray of string
- sql_taskobjectRequired
Show child attributesHide child attributes
- parametersobject
- run_job_taskobjectRequired
Show child attributesHide child attributes
- parametersobject
- job_parametersobject
- condition_taskobjectRequired
Show child attributesHide child attributes
- leftstring
- rightstring
- simulation_taskobjectRequired
Show child attributesHide child attributes
- parametersobject
- pipeline_taskobjectRequiredBeta
Show child attributesHide child attributes
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- descriptionstring
An optional description for this task.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. The key is
task_key, and the value is the name assigned to the dependent task.Show child attributesHide child attributes
- task_keystring
The name of the task this task depends on.
- outcomestring
Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.
- run_ifstring
An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, defaults to
ALL_SUCCESS. Seejobs/createfor a list of possible values.
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- email_notificationsobject
An optional set of email addresses notified when the task run begins or completes. The default behavior is to not send any emails.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
Show child attributesHide child attributes
- metricstring
- opstring
- valueint64
Specifies the threshold value that the health metric should obey to satisfy the health rule.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes. The default behavior is to not send any system notifications. Task webhooks respect the task notification settings.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.Show child attributesHide child attributes
- idstring
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.Show child attributesHide child attributes
- idstring
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.Show child attributesHide child attributes
- idstring
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.Show child attributesHide child attributes
- idstring
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.Show child attributesHide child attributes
- idstring
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
Show child attributesHide child attributes
- hardware_acceleratorstringBeta
Hardware accelerator configuration for Serverless GPU workloads.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.Show child attributesHide child attributes
- notebook_pathstring
The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
- base_parametersobject
Base parameters to be used for each run of this job. If the run is initiated by a call to
jobs/runNow with parameters specified, the two parameters maps are merged. If the same key is specified inbase_parametersand inrun-now, the value fromrun-nowis used. Use Task parameter variables to set parameters containing information about job runs.If the notebook takes a parameter that is not specified in the job’s
base_parametersor therun-nowoverride parameters, the default value from the notebook is used.Retrieve these parameters in a notebook using dbutils.widgets.get.
The JSON representation of this field cannot exceed 1MB.
- sourcestring
Optional location type of the notebook. When set to
WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set toGIT, the notebook will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Notebook is located in <Databricks> workspace.GIT: Notebook is located in cloud Git provider.
- warehouse_idstring
Optional
warehouse_idto run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.Show child attributesHide child attributes
- jar_uristring
Deprecated since 04/2016. For classic compute, provide a
jarthrough thelibrariesfield instead. For serverless compute, provide ajarthough thejava_dependenciesfield inside theenvironmentslist.See the examples of classic and serverless compute usage at the top of the page.
- main_class_namestring
The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.
The code must use
SparkContext.getOrCreateto obtain a Spark context; otherwise, runs of the job fail.
- parametersarray of string
Parameters passed to the main method.
Use Task parameter variables to set parameters containing information about job runs.
- run_as_replboolean
Deprecated. A value of
falseis no longer supported.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.Show child attributesHide child attributes
- python_filestring
The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with
/. For files stored in a remote repository, the path must be relative. This field is required.
- parametersarray of string
Command line parameters passed to the Python file.
Use Task parameter variables to set parameters containing information about job runs.
- sourcestring
Optional location type of the Python file. When set to
WORKSPACEor not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if thepython_filehas a URI format). When set toGIT, the Python file will be retrieved from a Git repository defined ingit_source.WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.GIT: The Python file is located in a remote Git repository.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
Show child attributesHide child attributes
- parametersarray of string
Command-line parameters passed to spark submit.
Use Task parameter variables to set parameters containing information about job runs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.Show child attributesHide child attributes
- pipeline_idstring
The full name of the pipeline task to execute.
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.Show child attributesHide child attributes
- package_namestring
Name of the package to execute
- entry_pointstring
Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using
$packageName.$entryPoint()
- parametersarray of string
Command-line parameters passed to Python wheel task. Leave it empty if
named_parametersis not null.
- named_parametersobject
Command-line parameters passed to Python wheel task in the form of
["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty ifparametersis not null.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.Show child attributesHide child attributes
- project_directorystring
Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.
- commandsarray of string
A list of dbt commands to execute. All commands must start with
dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.
- schemastring
Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the
defaultschema is used.
- warehouse_idstring
ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the
--profiles-dircommand line argument.
- profiles_directorystring
Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.
- catalogstring
Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
- sourcestring
Optional location type of the project directory. When set to
WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set toGIT, the project will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Project is located in <Databricks> workspace.GIT: Project is located in cloud Git provider.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.Show child attributesHide child attributes
- parametersobject
Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.
- queryobjectRequired
If query, indicates that this job must execute a SQL query.
Show child attributesHide child attributes
- query_idstringRequired
The canonical identifier of the SQL query.
- dashboardobjectRequired
If dashboard, indicates that this job must refresh a SQL dashboard.
Show child attributesHide child attributes
- dashboard_idstring
The canonical identifier of the SQL dashboard.
- subscriptionsarray of object
If specified, dashboard snapshots are sent to subscriptions.
- custom_subjectstring
Subject of the email sent to subscribers of this task.
- pause_subscriptionsboolean
If true, the dashboard snapshot is not taken, and emails are not sent to subscribers.
- alertobjectRequired
If alert, indicates that this job must refresh a SQL alert.
Show child attributesHide child attributes
- alert_idstring
The canonical identifier of the SQL alert.
- subscriptionsarray of object
If specified, alert notifications are sent to subscribers.
- pause_subscriptionsboolean
If true, the alert notifications are not sent to subscribers.
- fileobjectRequired
If file, indicates that this job runs a SQL file in a remote Git repository.
Show child attributesHide child attributes
- pathstring
Path of the SQL file. Must be relative if the source is a remote Git repository and absolute for workspace paths.
- sourcestring
Optional location type of the SQL file. When set to
WORKSPACE, the SQL file will be retrieved from the local <Databricks> workspace. When set toGIT, the SQL file will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: SQL file is located in <Databricks> workspace.GIT: SQL file is located in cloud Git provider.
- warehouse_idstring
The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.Show child attributesHide child attributes
- job_idint64
ID of the job to trigger.
- job_parametersobject
Job-level parameters used to trigger the job.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.Show child attributesHide child attributes
- opstring
EQUAL_TO,NOT_EQUALoperators perform string comparison of their operands. This means that“12.0” == “12”will evaluate tofalse.GREATER_THAN,GREATER_THAN_OR_EQUAL,LESS_THAN,LESS_THAN_OR_EQUALoperators perform numeric comparison of their operands.“12.0” >= “12”will evaluate totrue,“10.0” >= “12”will evaluate tofalse.
The boolean comparison to task values can be implemented with operators
EQUAL_TO,NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to“true”or“false”for the comparison.
- leftstring
The left operand of the condition task. Can be either a string value or a job state or parameter reference.
- rightstring
The right operand of the condition task. Can be either a string value or a job state or parameter reference.
- outcomestring
The condition expression evaluation result. Filled in if the task was successfully completed. Can be
"true"or"false"
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.Show child attributesHide child attributes
- inputsstring
Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
- concurrencyint32
An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
- taskobject
Configuration for the task that will be run for each element in the array
Show child attributesHide child attributes
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the
run_ifcondition is true. The key istask_key, and the value is the name assigned to the dependent task.
- run_ifstring
An optional value specifying the condition determining whether the task is run once its dependencies have been completed.
ALL_SUCCESS: All dependencies have executed and succeededAT_LEAST_ONE_SUCCESS: At least one dependency has succeededNONE_FAILED: None of the dependencies have failed and at least one was executedALL_DONE: All dependencies have been completedAT_LEAST_ONE_FAILED: At least one dependency failedALL_FAILED: ALl dependencies have failed
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- healthobject
- email_notificationsobject
An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task.
- webhook_notificationsobject
A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.
- descriptionstring
An optional description for this task.
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.Show child attributesHide child attributes
- clean_room_namestring
The clean room that the notebook belongs to.
- notebook_namestring
Name of the notebook being run.
- etagstring
Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the
cleanroomassets/getAPI.
- notebook_base_parametersobject
Base parameters to be used for the clean room notebook job.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.Show child attributesHide child attributes
- alert_idstringPublic Preview
The alert_id is the canonical identifier of the alert.
- warehouse_idstringPublic Preview
The warehouse_id identifies the warehouse settings used by the alert task.
- workspace_pathstringPublic Preview
The workspace_path is the path to the alert file in the workspace. The path:
- must start with "/Workspace"
- must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.
- subscribersarray of objectPublic Preview
The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.
Show child attributesHide child attributes
- user_namestringRequiredPublic Preview
A valid workspace email address.
- destination_idstringRequiredPublic Preview
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.Show child attributesHide child attributes
- tablesarray of objectPublic Preview
The tables to be exported to Power BI
Show child attributesHide child attributes
- namestringPublic Preview
The table name in <Databricks>
- catalogstringPublic Preview
The catalog name in <Databricks>
- schemastringPublic Preview
The schema name in <Databricks>
- storage_modestringPublic Preview
The Power BI storage mode of the table
- warehouse_idstringPublic Preview
The SQL warehouse ID to use as the Power BI data source
- power_bi_modelobjectPublic Preview
The semantic model to update
Show child attributesHide child attributes
- workspace_namestringPublic Preview
The name of the Power BI workspace of the model
- model_namestringPublic Preview
The name of the Power BI model
- storage_modestringPublic Preview
The default storage mode of the Power BI model
- authentication_methodstringPublic Preview
How the published Power BI model authenticates to <Databricks>
- overwrite_existingbooleanPublic Preview
Whether to overwrite existing Power BI models
- connection_resource_namestringPublic Preview
The resource name of the UC connection to authenticate from <Databricks> to Power BI
- refresh_after_updatebooleanPublic Preview
Whether the model should be refreshed after the update
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
Show child attributesHide child attributes
- subscriptionobject
Optional: subscription configuration for sending the dashboard snapshot.
Show child attributesHide child attributes
- subscribersarray of object
The list of subscribers to send the snapshot of the dashboard to.
- pausedboolean
When true, the subscription will not send emails.
- custom_subjectstring
Optional: Allows users to specify a custom subject line on the email sent to subscribers.
- warehouse_idstring
Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
- dashboard_idstring
The identifier of the dashboard to refresh.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
Show child attributesHide child attributes
- experimentstringPublic Preview
MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set
mlflow_experiment_directory.
- deploymentsarray of objectPublic Preview
Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.
Show child attributesHide child attributes
- command_pathstringPublic Preview
Workspace path of the script to run on each node in this deployment. Upload the script to this path and supply the path here. When the task runs, the file at this path is run on each node; if it fails, the task fails with its exit code.
Example script contents:
Plain Python:
python train.py --epochs 10
Multi-GPU via accelerate:
accelerate launch train.py --config config.yaml
Distributed via torchrun:
torchrun --nproc_per_node=8 train.py
- computeobjectPublic Preview
Compute resources allocated to each node in this deployment.
- namestringPublic Preview
Optional human-readable name for this deployment (for example,
driver,worker,param_server). Used for log and UI display. Distinct names are recommended so deployments can be told apart, but uniqueness is not enforced.
- mlflow_runstringPublic Preview
Optional display name for the MLflow run created under
experiment. If omitted, MLflow generates a default name.
- mlflow_experiment_directorystringPublic Preview
Optional workspace directory under which the MLflow experiment named in
experimentis created. Must start with/Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
- docker_image_urlstringBeta
Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format:
{organization}/{repository}:{tag}
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstring
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectRequired
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobjectRequired
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobjectRequired
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobjectRequired
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobjectRequired
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobjectRequired
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobjectRequired
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }Show child attributesHide child attributes
- packagestring
The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".
- repostring
The repository where the package can be found. If not specified, the default pip index is used.
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }Show child attributesHide child attributes
- coordinatesstring
Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
- repostring
Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.
- exclusionsarray of string
List of dependences to exclude. For example:
["slf4j:slf4j", "*:hadoop-client"].Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
Show child attributesHide child attributes
- packagestring
The name of the CRAN package to install.
- repostring
The repository where the package can be found. If not specified, the default CRAN repo is used.
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- descriptionstring
Description of the run
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- job_clustersarray of object
A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings. If more than 100 job clusters are available, you can paginate through them using
jobs/getrun.Show child attributesHide child attributes
- job_cluster_keystring
A unique name for the job cluster. This field is required and must be unique within the job.
JobTaskSettingsmay refer to this field to determine which cluster to launch for the task execution.
- new_clusterobject
If new_cluster, a description of a cluster that is created for each task.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstring
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectRequired
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobjectRequired
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobjectRequired
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobjectRequired
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobjectRequired
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobjectRequired
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobjectRequired
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.
If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task.Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,
git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobject
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- repair_historyarray of object
The repair history of the run.
Show child attributesHide child attributes
- typestring
The repair history item type. Indicates whether a run is the original run or a repair run.
- start_timeint64
The start time of the (repaired) run.
- end_timeint64
The end time of the (repaired) run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- idint64
The ID of the repair. Only returned for the items that represent a repair in
repair_history.
- task_run_idsarray of int64
The run IDs of the task runs that ran as part of this repair history item.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- job_run_idint64
ID of the job run that this run belongs to. For legacy and single-task job runs the field is populated with the job run ID. For task runs, the field is populated with the ID of the job run that the task run belongs to.
- has_moreboolean
Indicates if the run has more array properties (
tasks,job_clusters) that are not shown. They can be accessed viajobs/getrunendpoint. It is only relevant for API 2.2jobs/listrunsrequests withexpand_tasks=true.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
List GA
GET
List runs in descending order by end time. If a run has not finished, it falls back to start time.
API scopes: jobs
Parameters
- job_idint64query
The job for which to list runs. If omitted, the Jobs service lists runs from all jobs.
- active_onlybooleanRequiredquery
If active_only is
true, only active runs are included in the results; otherwise, lists both active and completed runs. An active run is a run in theQUEUED,PENDING,RUNNING, orTERMINATING. This field cannot betruewhen completed_only istrue.
- completed_onlybooleanRequiredquery
If completed_only is
true, only completed runs are included in the results; otherwise, lists both active and completed runs. This field cannot betruewhen active_only istrue.
- limitint32query
The number of runs to return. This value must be greater than 0 and less than 25. The default value is 20. If a request specifies a limit of 0, the service instead uses the maximum limit.
- run_typestringquery
The type of runs to return. For a description of run types, see
jobs/getRun.
- expand_tasksbooleanquery
Whether to include task and cluster details in the response. Note that only the first 100 elements will be shown. Use
jobs/getrunto paginate through all tasks and clusters.
- start_time_fromint64query
Show runs that started at or after this value. The value must be a UTC timestamp in milliseconds. Can be combined with start_time_to to filter by a time range.
- start_time_toint64query
Show runs that started at or before this value. The value must be a UTC timestamp in milliseconds. Can be combined with start_time_from to filter by a time range.
- page_tokenstringquery
Use
next_page_tokenorprev_page_tokenreturned from the previous request to list the next or previous page of runs respectively.
Response
- runsarray of object
A list of runs, from most recently started to least. Only included in the response if there are runs to list.
Show child attributesHide child attributes
- job_idint64
The canonical identifier of the job that contains this run.
- run_idint64
The canonical identifier of the run. This ID is unique across all runs of all jobs.
- creator_user_namestring
The creator user name. This field won’t be included in the response if the user has already been deleted.
- number_in_jobint64
A unique identifier for this job run. This is set to the same value as
run_id.
- original_attempt_run_idint64
If this run is a retry of a prior run attempt, this field contains the run_id of the original attempt; otherwise, it is the same as the run_id.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- scheduleobject
The cron schedule that triggered this run if it was triggered by the periodic scheduler.
Show child attributesHide child attributes
- quartz_cron_expressionstring
A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
- timezone_idstring
A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
- pause_statusstring
Indicate whether this schedule is paused or not.
- cluster_specobject
A snapshot of the job’s cluster specification when this run was created.
Show child attributesHide child attributes
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- job_parametersarray of object
Job-level parameters used in the run
Show child attributesHide child attributes
- namestring
The name of the parameter
- defaultstring
The optional default value of the parameter
- valuestring
The value used in the run
- overriding_parametersobject
The parameters used for this run.
Show child attributesHide child attributes
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- triggerstring
- trigger_infoobject
Show child attributesHide child attributes
- run_idint64
The run id of the Run Job task run
- run_namestring
An optional name for the run. The maximum length is 4096 bytes in UTF-8 encoding.
- run_page_urlstring
The URL to the detail page of the run.
- run_typestring
- tasksarray of object
The list of tasks performed by the run. Each task has its own
run_idwhich you can use to callJobsGetOutputto retrieve the run results. If more than 100 tasks are available, you can paginate through them usingjobs/getrun. Use thenext_page_tokenfield at the object root to determine if more results are available.Show child attributesHide child attributes
- run_idint64
The ID of the task run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- run_page_urlstring
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
- sparse_checkoutobject
- resolved_valuesobject
Parameter values including resolved references
Show child attributesHide child attributes
- notebook_taskobjectRequired
- spark_jar_taskobjectRequired
- spark_python_taskobjectRequired
- spark_submit_taskobjectRequired
- python_wheel_taskobjectRequired
- dbt_taskobjectRequired
- sql_taskobjectRequired
- run_job_taskobjectRequired
- condition_taskobjectRequired
- simulation_taskobjectRequired
- pipeline_taskobjectRequiredBeta
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- descriptionstring
An optional description for this task.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. The key is
task_key, and the value is the name assigned to the dependent task.Show child attributesHide child attributes
- task_keystring
The name of the task this task depends on.
- outcomestring
Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.
- run_ifstring
An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, defaults to
ALL_SUCCESS. Seejobs/createfor a list of possible values.
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- email_notificationsobject
An optional set of email addresses notified when the task run begins or completes. The default behavior is to not send any emails.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes. The default behavior is to not send any system notifications. Task webhooks respect the task notification settings.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
Show child attributesHide child attributes
- hardware_acceleratorstringBeta
Hardware accelerator configuration for Serverless GPU workloads.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.Show child attributesHide child attributes
- notebook_pathstring
The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
- base_parametersobject
Base parameters to be used for each run of this job. If the run is initiated by a call to
jobs/runNow with parameters specified, the two parameters maps are merged. If the same key is specified inbase_parametersand inrun-now, the value fromrun-nowis used. Use Task parameter variables to set parameters containing information about job runs.If the notebook takes a parameter that is not specified in the job’s
base_parametersor therun-nowoverride parameters, the default value from the notebook is used.Retrieve these parameters in a notebook using dbutils.widgets.get.
The JSON representation of this field cannot exceed 1MB.
- sourcestring
Optional location type of the notebook. When set to
WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set toGIT, the notebook will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Notebook is located in <Databricks> workspace.GIT: Notebook is located in cloud Git provider.
- warehouse_idstring
Optional
warehouse_idto run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.Show child attributesHide child attributes
- jar_uristring
Deprecated since 04/2016. For classic compute, provide a
jarthrough thelibrariesfield instead. For serverless compute, provide ajarthough thejava_dependenciesfield inside theenvironmentslist.See the examples of classic and serverless compute usage at the top of the page.
- main_class_namestring
The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.
The code must use
SparkContext.getOrCreateto obtain a Spark context; otherwise, runs of the job fail.
- parametersarray of string
Parameters passed to the main method.
Use Task parameter variables to set parameters containing information about job runs.
- run_as_replboolean
Deprecated. A value of
falseis no longer supported.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.Show child attributesHide child attributes
- python_filestring
The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with
/. For files stored in a remote repository, the path must be relative. This field is required.
- parametersarray of string
Command line parameters passed to the Python file.
Use Task parameter variables to set parameters containing information about job runs.
- sourcestring
Optional location type of the Python file. When set to
WORKSPACEor not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if thepython_filehas a URI format). When set toGIT, the Python file will be retrieved from a Git repository defined ingit_source.WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.GIT: The Python file is located in a remote Git repository.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
Show child attributesHide child attributes
- parametersarray of string
Command-line parameters passed to spark submit.
Use Task parameter variables to set parameters containing information about job runs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.Show child attributesHide child attributes
- pipeline_idstring
The full name of the pipeline task to execute.
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.Show child attributesHide child attributes
- package_namestring
Name of the package to execute
- entry_pointstring
Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using
$packageName.$entryPoint()
- parametersarray of string
Command-line parameters passed to Python wheel task. Leave it empty if
named_parametersis not null.
- named_parametersobject
Command-line parameters passed to Python wheel task in the form of
["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty ifparametersis not null.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.Show child attributesHide child attributes
- project_directorystring
Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.
- commandsarray of string
A list of dbt commands to execute. All commands must start with
dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.
- schemastring
Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the
defaultschema is used.
- warehouse_idstring
ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the
--profiles-dircommand line argument.
- profiles_directorystring
Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.
- catalogstring
Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
- sourcestring
Optional location type of the project directory. When set to
WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set toGIT, the project will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Project is located in <Databricks> workspace.GIT: Project is located in cloud Git provider.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.Show child attributesHide child attributes
- parametersobject
Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.
- queryobjectRequired
If query, indicates that this job must execute a SQL query.
- dashboardobjectRequired
If dashboard, indicates that this job must refresh a SQL dashboard.
- alertobjectRequired
If alert, indicates that this job must refresh a SQL alert.
- fileobjectRequired
If file, indicates that this job runs a SQL file in a remote Git repository.
- warehouse_idstring
The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.Show child attributesHide child attributes
- job_idint64
ID of the job to trigger.
- job_parametersobject
Job-level parameters used to trigger the job.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.Show child attributesHide child attributes
- opstring
EQUAL_TO,NOT_EQUALoperators perform string comparison of their operands. This means that“12.0” == “12”will evaluate tofalse.GREATER_THAN,GREATER_THAN_OR_EQUAL,LESS_THAN,LESS_THAN_OR_EQUALoperators perform numeric comparison of their operands.“12.0” >= “12”will evaluate totrue,“10.0” >= “12”will evaluate tofalse.
The boolean comparison to task values can be implemented with operators
EQUAL_TO,NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to“true”or“false”for the comparison.
- leftstring
The left operand of the condition task. Can be either a string value or a job state or parameter reference.
- rightstring
The right operand of the condition task. Can be either a string value or a job state or parameter reference.
- outcomestring
The condition expression evaluation result. Filled in if the task was successfully completed. Can be
"true"or"false"
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.Show child attributesHide child attributes
- inputsstring
Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
- concurrencyint32
An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
- taskobject
Configuration for the task that will be run for each element in the array
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.Show child attributesHide child attributes
- clean_room_namestring
The clean room that the notebook belongs to.
- notebook_namestring
Name of the notebook being run.
- etagstring
Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the
cleanroomassets/getAPI.
- notebook_base_parametersobject
Base parameters to be used for the clean room notebook job.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.Show child attributesHide child attributes
- alert_idstringPublic Preview
The alert_id is the canonical identifier of the alert.
- warehouse_idstringPublic Preview
The warehouse_id identifies the warehouse settings used by the alert task.
- workspace_pathstringPublic Preview
The workspace_path is the path to the alert file in the workspace. The path:
- must start with "/Workspace"
- must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.
- subscribersarray of objectPublic Preview
The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.Show child attributesHide child attributes
- tablesarray of objectPublic Preview
The tables to be exported to Power BI
- warehouse_idstringPublic Preview
The SQL warehouse ID to use as the Power BI data source
- power_bi_modelobjectPublic Preview
The semantic model to update
- connection_resource_namestringPublic Preview
The resource name of the UC connection to authenticate from <Databricks> to Power BI
- refresh_after_updatebooleanPublic Preview
Whether the model should be refreshed after the update
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
Show child attributesHide child attributes
- subscriptionobject
Optional: subscription configuration for sending the dashboard snapshot.
- warehouse_idstring
Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
- dashboard_idstring
The identifier of the dashboard to refresh.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
Show child attributesHide child attributes
- experimentstringPublic Preview
MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set
mlflow_experiment_directory.
- deploymentsarray of objectPublic Preview
Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.
- mlflow_runstringPublic Preview
Optional display name for the MLflow run created under
experiment. If omitted, MLflow generates a default name.
- mlflow_experiment_directorystringPublic Preview
Optional workspace directory under which the MLflow experiment named in
experimentis created. Must start with/Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
- docker_image_urlstringBeta
Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format:
{organization}/{repository}:{tag}
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- descriptionstring
Description of the run
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- job_clustersarray of object
A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings. If more than 100 job clusters are available, you can paginate through them using
jobs/getrun.Show child attributesHide child attributes
- job_cluster_keystring
A unique name for the job cluster. This field is required and must be unique within the job.
JobTaskSettingsmay refer to this field to determine which cluster to launch for the task execution.
- new_clusterobject
If new_cluster, a description of a cluster that is created for each task.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.
If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task.Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,
git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobject
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- repair_historyarray of object
The repair history of the run.
Show child attributesHide child attributes
- typestring
The repair history item type. Indicates whether a run is the original run or a repair run.
- start_timeint64
The start time of the (repaired) run.
- end_timeint64
The end time of the (repaired) run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- idint64
The ID of the repair. Only returned for the items that represent a repair in
repair_history.
- task_run_idsarray of int64
The run IDs of the task runs that ran as part of this repair history item.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- job_run_idint64
ID of the job run that this run belongs to. For legacy and single-task job runs the field is populated with the job run ID. For task runs, the field is populated with the ID of the job run that the task run belongs to.
- has_moreboolean
Indicates if the run has more array properties (
tasks,job_clusters) that are not shown. They can be accessed viajobs/getrunendpoint. It is only relevant for API 2.2jobs/listrunsrequests withexpand_tasks=true.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- next_page_tokenstring
A token that can be used to list the next page of runs (if applicable).
- prev_page_tokenstring
A token that can be used to list the previous page of runs (if applicable).
Delete GA
Submit Run GA
POST
Submit a one-time run. This endpoint allows you to submit a workload directly
without creating a job. Runs submitted using this endpoint don’t display in
the UI. Use the jobs/runs/get API to check the run state after the job is
submitted.
Important: Jobs submitted using this endpoint are not saved as a job.
They do not show up in the Jobs UI, and do not retry when they fail. Because
they are not saved, <Databricks> cannot auto-optimize serverless compute in case
of failure. If your job fails, you may want to use classic compute to specify
the compute needs for the job. Alternatively, use the POST /jobs/create and
POST /jobs/run-now endpoints to create and run a saved job.
API scopes: jobs
Request body
- access_control_listarray of object
List of permissions to set on the job.
Show child attributesHide child attributes
- user_namestringRequired
- group_namestringRequired
- service_principal_namestringRequired
- permission_levelstring
- queueobject
The queue settings of the one-time run.
Show child attributesHide child attributes
- enabledboolean
If true, enable queueing for the job. This is a required field.
- run_asobject
Specifies the user or service principal that the job runs as. If not specified, the job runs as the user who submits the request.
Show child attributesHide child attributes
- user_namestringRequired
The email of an active workspace user. Non-admin users can only set this field to their own email.
- service_principal_namestringRequired
Application ID of an active service principal. Setting this field requires the
servicePrincipal/userrole.
- run_namestring
An optional name for the run. The default value is
Untitled.
- timeout_secondsint32
An optional timeout applied to each run of this job. A value of
0means no timeout.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
Show child attributesHide child attributes
- metricstring
- opstring
- valueint64
Specifies the threshold value that the health metric should obey to satisfy the health rule.
- idempotency_tokenstring
An optional token that can be used to guarantee the idempotency of job run requests. If a run with the provided token already exists, the request does not create a new run but returns the ID of the existing run instead. If a run with the provided token is deleted, an error is returned.
If you specify the idempotency token, upon failure you can retry until the request succeeds. <Databricks> guarantees that exactly one run is launched with that idempotency token.
This token must have at most 64 characters.
- tasksarray of object
Show child attributesHide child attributes
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- descriptionstring
An optional description for this task.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. The key is
task_key, and the value is the name assigned to the dependent task.Show child attributesHide child attributes
- task_keystring
The name of the task this task depends on.
- outcomestring
Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.
- run_ifstring
An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, defaults to
ALL_SUCCESS. Seejobs/createfor a list of possible values.
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- email_notificationsobject
An optional set of email addresses notified when the task run begins or completes. The default behavior is to not send any emails.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
Show child attributesHide child attributes
- metricstring
- opstring
- valueint64
Specifies the threshold value that the health metric should obey to satisfy the health rule.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes. The default behavior is to not send any system notifications. Task webhooks respect the task notification settings.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.Show child attributesHide child attributes
- idstring
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.Show child attributesHide child attributes
- idstring
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.Show child attributesHide child attributes
- idstring
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.Show child attributesHide child attributes
- idstring
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.Show child attributesHide child attributes
- idstring
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
Show child attributesHide child attributes
- hardware_acceleratorstringBeta
Hardware accelerator configuration for Serverless GPU workloads.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.Show child attributesHide child attributes
- notebook_pathstring
The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
- base_parametersobject
Base parameters to be used for each run of this job. If the run is initiated by a call to
jobs/runNow with parameters specified, the two parameters maps are merged. If the same key is specified inbase_parametersand inrun-now, the value fromrun-nowis used. Use Task parameter variables to set parameters containing information about job runs.If the notebook takes a parameter that is not specified in the job’s
base_parametersor therun-nowoverride parameters, the default value from the notebook is used.Retrieve these parameters in a notebook using dbutils.widgets.get.
The JSON representation of this field cannot exceed 1MB.
- sourcestring
Optional location type of the notebook. When set to
WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set toGIT, the notebook will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Notebook is located in <Databricks> workspace.GIT: Notebook is located in cloud Git provider.
- warehouse_idstring
Optional
warehouse_idto run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.Show child attributesHide child attributes
- jar_uristring
Deprecated since 04/2016. For classic compute, provide a
jarthrough thelibrariesfield instead. For serverless compute, provide ajarthough thejava_dependenciesfield inside theenvironmentslist.See the examples of classic and serverless compute usage at the top of the page.
- main_class_namestring
The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.
The code must use
SparkContext.getOrCreateto obtain a Spark context; otherwise, runs of the job fail.
- parametersarray of string
Parameters passed to the main method.
Use Task parameter variables to set parameters containing information about job runs.
- run_as_replboolean
Deprecated. A value of
falseis no longer supported.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.Show child attributesHide child attributes
- python_filestring
The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with
/. For files stored in a remote repository, the path must be relative. This field is required.
- parametersarray of string
Command line parameters passed to the Python file.
Use Task parameter variables to set parameters containing information about job runs.
- sourcestring
Optional location type of the Python file. When set to
WORKSPACEor not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if thepython_filehas a URI format). When set toGIT, the Python file will be retrieved from a Git repository defined ingit_source.WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.GIT: The Python file is located in a remote Git repository.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
Show child attributesHide child attributes
- parametersarray of string
Command-line parameters passed to spark submit.
Use Task parameter variables to set parameters containing information about job runs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.Show child attributesHide child attributes
- pipeline_idstring
The full name of the pipeline task to execute.
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.Show child attributesHide child attributes
- package_namestring
Name of the package to execute
- entry_pointstring
Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using
$packageName.$entryPoint()
- parametersarray of string
Command-line parameters passed to Python wheel task. Leave it empty if
named_parametersis not null.
- named_parametersobject
Command-line parameters passed to Python wheel task in the form of
["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty ifparametersis not null.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.Show child attributesHide child attributes
- project_directorystring
Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.
- commandsarray of string
A list of dbt commands to execute. All commands must start with
dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.
- schemastring
Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the
defaultschema is used.
- warehouse_idstring
ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the
--profiles-dircommand line argument.
- profiles_directorystring
Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.
- catalogstring
Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
- sourcestring
Optional location type of the project directory. When set to
WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set toGIT, the project will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Project is located in <Databricks> workspace.GIT: Project is located in cloud Git provider.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.Show child attributesHide child attributes
- parametersobject
Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.
- queryobjectRequired
If query, indicates that this job must execute a SQL query.
Show child attributesHide child attributes
- query_idstringRequired
The canonical identifier of the SQL query.
- dashboardobjectRequired
If dashboard, indicates that this job must refresh a SQL dashboard.
Show child attributesHide child attributes
- dashboard_idstring
The canonical identifier of the SQL dashboard.
- subscriptionsarray of object
If specified, dashboard snapshots are sent to subscriptions.
- custom_subjectstring
Subject of the email sent to subscribers of this task.
- pause_subscriptionsboolean
If true, the dashboard snapshot is not taken, and emails are not sent to subscribers.
- alertobjectRequired
If alert, indicates that this job must refresh a SQL alert.
Show child attributesHide child attributes
- alert_idstring
The canonical identifier of the SQL alert.
- subscriptionsarray of object
If specified, alert notifications are sent to subscribers.
- pause_subscriptionsboolean
If true, the alert notifications are not sent to subscribers.
- fileobjectRequired
If file, indicates that this job runs a SQL file in a remote Git repository.
Show child attributesHide child attributes
- pathstring
Path of the SQL file. Must be relative if the source is a remote Git repository and absolute for workspace paths.
- sourcestring
Optional location type of the SQL file. When set to
WORKSPACE, the SQL file will be retrieved from the local <Databricks> workspace. When set toGIT, the SQL file will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: SQL file is located in <Databricks> workspace.GIT: SQL file is located in cloud Git provider.
- warehouse_idstring
The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.Show child attributesHide child attributes
- job_idint64
ID of the job to trigger.
- job_parametersobject
Job-level parameters used to trigger the job.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.Show child attributesHide child attributes
- opstring
EQUAL_TO,NOT_EQUALoperators perform string comparison of their operands. This means that“12.0” == “12”will evaluate tofalse.GREATER_THAN,GREATER_THAN_OR_EQUAL,LESS_THAN,LESS_THAN_OR_EQUALoperators perform numeric comparison of their operands.“12.0” >= “12”will evaluate totrue,“10.0” >= “12”will evaluate tofalse.
The boolean comparison to task values can be implemented with operators
EQUAL_TO,NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to“true”or“false”for the comparison.
- leftstring
The left operand of the condition task. Can be either a string value or a job state or parameter reference.
- rightstring
The right operand of the condition task. Can be either a string value or a job state or parameter reference.
- outcomestring
The condition expression evaluation result. Filled in if the task was successfully completed. Can be
"true"or"false"
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.Show child attributesHide child attributes
- inputsstring
Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
- concurrencyint32
An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
- taskobject
Configuration for the task that will be run for each element in the array
Show child attributesHide child attributes
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the
run_ifcondition is true. The key istask_key, and the value is the name assigned to the dependent task.
- run_ifstring
An optional value specifying the condition determining whether the task is run once its dependencies have been completed.
ALL_SUCCESS: All dependencies have executed and succeededAT_LEAST_ONE_SUCCESS: At least one dependency has succeededNONE_FAILED: None of the dependencies have failed and at least one was executedALL_DONE: All dependencies have been completedAT_LEAST_ONE_FAILED: At least one dependency failedALL_FAILED: ALl dependencies have failed
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- healthobject
- email_notificationsobject
An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task.
- webhook_notificationsobject
A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.
- descriptionstring
An optional description for this task.
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.Show child attributesHide child attributes
- clean_room_namestring
The clean room that the notebook belongs to.
- notebook_namestring
Name of the notebook being run.
- etagstring
Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the
cleanroomassets/getAPI.
- notebook_base_parametersobject
Base parameters to be used for the clean room notebook job.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.Show child attributesHide child attributes
- alert_idstringPublic Preview
The alert_id is the canonical identifier of the alert.
- warehouse_idstringPublic Preview
The warehouse_id identifies the warehouse settings used by the alert task.
- workspace_pathstringPublic Preview
The workspace_path is the path to the alert file in the workspace. The path:
- must start with "/Workspace"
- must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.
- subscribersarray of objectPublic Preview
The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.
Show child attributesHide child attributes
- user_namestringRequiredPublic Preview
A valid workspace email address.
- destination_idstringRequiredPublic Preview
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.Show child attributesHide child attributes
- tablesarray of objectPublic Preview
The tables to be exported to Power BI
Show child attributesHide child attributes
- namestringPublic Preview
The table name in <Databricks>
- catalogstringPublic Preview
The catalog name in <Databricks>
- schemastringPublic Preview
The schema name in <Databricks>
- storage_modestringPublic Preview
The Power BI storage mode of the table
- warehouse_idstringPublic Preview
The SQL warehouse ID to use as the Power BI data source
- power_bi_modelobjectPublic Preview
The semantic model to update
Show child attributesHide child attributes
- workspace_namestringPublic Preview
The name of the Power BI workspace of the model
- model_namestringPublic Preview
The name of the Power BI model
- storage_modestringPublic Preview
The default storage mode of the Power BI model
- authentication_methodstringPublic Preview
How the published Power BI model authenticates to <Databricks>
- overwrite_existingbooleanPublic Preview
Whether to overwrite existing Power BI models
- connection_resource_namestringPublic Preview
The resource name of the UC connection to authenticate from <Databricks> to Power BI
- refresh_after_updatebooleanPublic Preview
Whether the model should be refreshed after the update
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
Show child attributesHide child attributes
- subscriptionobject
Optional: subscription configuration for sending the dashboard snapshot.
Show child attributesHide child attributes
- subscribersarray of object
The list of subscribers to send the snapshot of the dashboard to.
- pausedboolean
When true, the subscription will not send emails.
- custom_subjectstring
Optional: Allows users to specify a custom subject line on the email sent to subscribers.
- warehouse_idstring
Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
- dashboard_idstring
The identifier of the dashboard to refresh.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
Show child attributesHide child attributes
- experimentstringPublic Preview
MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set
mlflow_experiment_directory.
- deploymentsarray of objectPublic Preview
Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.
Show child attributesHide child attributes
- command_pathstringPublic Preview
Workspace path of the script to run on each node in this deployment. Upload the script to this path and supply the path here. When the task runs, the file at this path is run on each node; if it fails, the task fails with its exit code.
Example script contents:
Plain Python:
python train.py --epochs 10
Multi-GPU via accelerate:
accelerate launch train.py --config config.yaml
Distributed via torchrun:
torchrun --nproc_per_node=8 train.py
- computeobjectPublic Preview
Compute resources allocated to each node in this deployment.
- namestringPublic Preview
Optional human-readable name for this deployment (for example,
driver,worker,param_server). Used for log and UI display. Distinct names are recommended so deployments can be told apart, but uniqueness is not enforced.
- mlflow_runstringPublic Preview
Optional display name for the MLflow run created under
experiment. If omitted, MLflow generates a default name.
- mlflow_experiment_directorystringPublic Preview
Optional workspace directory under which the MLflow experiment named in
experimentis created. Must start with/Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
- docker_image_urlstringBeta
Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format:
{organization}/{repository}:{tag}
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
- zone_idstring
Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.
The list of available zones as well as the default value can be found by using the
List Zonesmethod.
- instance_profile_arnstring
Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.
This feature may only be available to certain customer plans.
- spot_bid_price_percentint32
The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new
r3.xlargespot instance, then the bid price is half of the price of on-demandr3.xlargeinstances. Similarly, if this field is set to 200, the bid price is twice the price of on-demandr3.xlargeinstances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
- ebs_volume_typestring
The type of EBS volumes that will be launched with this cluster.
- ebs_volume_countint32
The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.
These EBS volumes will be mounted at
/ebs0,/ebs1, and etc. Instance store volumes will be mounted at/local_disk0,/local_disk1, and etc.If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.
Please note that if EBS volumes are specified, then the Spark configuration
spark.local.dirwill be overridden.
- ebs_volume_sizeint32
The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.
- ebs_volume_iopsint32
If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- ebs_volume_throughputint32
If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- log_analytics_infoobject
Defines values necessary to configure and run Azure Log Analytics agent
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- availabilitystring
Availability type used for all subsequent nodes past the
first_on_demandones. Note: Iffirst_on_demandis zero, this availability type will be used for the entire cluster.
- spot_bid_max_pricedouble
The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
- capacity_reservation_groupstring
The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.
Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:
- Microsoft.Compute/capacityReservationGroups/read
- Microsoft.Compute/capacityReservationGroups/deploy/action
- Microsoft.Compute/capacityReservationGroups/capacityReservations/read
- Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action
Format:
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
Show child attributesHide child attributes
- use_preemptible_executorsboolean
This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
- google_service_accountstring
If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.
- boot_disk_sizeint32
Boot disk size in GB
- availabilitystring
This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
- zone_idstring
Identifier for the availability zone in which the cluster resides. This can be one of the following:
- "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
- "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
- A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
- local_ssd_countint32
If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.
- first_on_demandint32
The first
first_on_demandnodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size,first_on_demandnodes will be placed on on-demand instances and the remainder will be placed onavailabilityinstances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
Show child attributesHide child attributes
- alternate_node_type_idsarray of string
A list of node type IDs to use as fallbacks when the primary node type is unavailable.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- volumesobjectRequired
destination needs to be provided, e.g.
{ "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.Show child attributesHide child attributes
- dbfsobjectRequired
destination needs to be provided. e.g.
{ "dbfs": { "destination" : "dbfs:/home/cluster_log" } }
- s3objectRequired
destination and either the region or endpoint need to be provided. e.g.
{ \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } }Cluster iam role is used to access s3, please make sure the cluster iam role ininstance_profile_arnhas permission to write data to the s3 destination.
- fileobjectRequired
destination needs to be provided, e.g.
{ "file": { "destination": "file:/my/local/file.sh" } }
- gcsobjectRequired
destination needs to be provided, e.g.
{ "gcs": { "destination": "gs://my-bucket/file.sh" } }
- abfssobjectRequired
destination needs to be provided, e.g.
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>
- workspaceobjectRequired
destination needs to be provided, e.g.
{ "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }
- volumesobjectRequired
destination needs to be provided. e.g.
{ \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }
- docker_imageobject
Custom docker image BYOC
Show child attributesHide child attributes
- urlstring
URL of the docker image.
- basic_authobjectRequired
Basic auth with username and password
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
Show child attributesHide child attributes
- clientsobject
defined what type of clients can use the cluster. E.g. Notebooks, Jobs
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
Show child attributesHide child attributes
- min_workersint32
The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
- max_workersint32
The maximum number of workers to which the cluster can scale up when overloaded. Note that
max_workersmust be strictly greater thanmin_workers.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }Show child attributesHide child attributes
- packagestring
The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".
- repostring
The repository where the package can be found. If not specified, the default pip index is used.
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }Show child attributesHide child attributes
- coordinatesstring
Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
- repostring
Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.
- exclusionsarray of string
List of dependences to exclude. For example:
["slf4j:slf4j", "*:hadoop-client"].Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
Show child attributesHide child attributes
- packagestring
The name of the CRAN package to install.
- repostring
The repository where the package can be found. If not specified, the default CRAN repo is used.
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.
If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task.Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,
git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobject
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.Show child attributesHide child attributes
- idstring
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.Show child attributesHide child attributes
- idstring
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.Show child attributesHide child attributes
- idstring
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.Show child attributesHide child attributes
- idstring
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.Show child attributesHide child attributes
- idstring
- email_notificationsobject
An optional set of email addresses notified when the run begins or completes.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- environmentsarray of object
A list of task execution environment specifications that can be referenced by tasks of this run.
Show child attributesHide child attributes
- environment_keystring
The key of an environment. It has to be unique within a job.
- specobject
Show child attributesHide child attributes
- clientstring
Use
environment_versioninstead.
- dependenciesarray of string
List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a valid pip requirements file line per https://pip.pypa.io/en/stable/reference/requirements-file-format/. Allowed dependencies include a requirement specifier, an archive URL, a local project path (such as WSFS or UC Volumes in <Databricks>), or a VCS project URL.
- base_environmentstring
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom
env.yamlfile (e.g.,/Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g.,workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g.,workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Eitherenvironment_versionorbase_environmentcan be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, seeAWSThe base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom
env.yamlfile (e.g.,/Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g.,workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g.,workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Eitherenvironment_versionorbase_environmentcan be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see https://docs.databricks.com/aws/en/compute/serverless/dependencies#base-environments-for-job-tasks.AzureThe base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom
env.yamlfile (e.g.,/Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g.,workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g.,workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Eitherenvironment_versionorbase_environmentcan be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, seeGCPThe base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom
env.yamlfile (e.g.,/Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g.,workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g.,workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Eitherenvironment_versionorbase_environmentcan be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, seehttps://docs.databricks.com/gcp/en/compute/serverless/dependencies#base-environments-for-job-tasks.
- environment_versionstring
Either
environment_versionorbase_environmentneeds to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.AWSEither
environment_versionorbase_environmentneeds to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. See https://docs.databricks.com/aws/release-notes/serverless/#serverless-environment-versions.AzureEither
environment_versionorbase_environmentneeds to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.GCPEither
environment_versionorbase_environmentneeds to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.See https://docs.databricks.com/gcp/release-notes/serverless/#serverless-environment-versions.
- java_dependenciesarray of string
List of java dependencies. Each dependency is a string representing a java library path. For example:
/Volumes/path/to/test.jar.AWSList of java dependencies. Each dependency is a string representing a java library path. For example:
/Volumes/path/to/test.jar. See https://docs.databricks.com/aws/en/jobs/jar.AzureList of java dependencies. Each dependency is a string representing a java library path. For example:
/Volumes/path/to/test.jar.See https://learn.microsoft.com/en-gb/azure/databricks/jobs/jar.
GCPList of java dependencies. Each dependency is a string representing a java library path. For example:
/Volumes/path/to/test.jar.
- budget_policy_idstringPublic Preview
The user specified id of the budget policy to use for this one-time run. If not specified, the run will be not be attributed to any budget policy.
- performance_targetstring
The performance mode on a serverless one-time run. This field determines the level of compute performance or cost-efficiency for the run. The performance target does not apply to tasks that run on Serverless GPU compute.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
Response
- run_idint64
The canonical identifier for the newly submitted run.
Repair GA
POST
Re-run one or more tasks. Tasks are re-run as part of the original job run. They use the current job and task settings, and can be viewed in the history for the original job run.
API scopes: jobs
Request body
- run_idint64
The job run ID of the run to repair. The run must not be in progress.
- latest_repair_idint64
The ID of the latest repair. This parameter is not required when repairing a run for the first time, but must be provided on subsequent requests to repair the same run.
- rerun_tasksarray of string
The task keys of the task runs to repair.
- job_parametersobject
Job-level parameters used in the run. for example
"param": "overriding_val"
- rerun_all_failed_tasksboolean
If true, repair all failed tasks. Only one of
rerun_tasksorrerun_all_failed_taskscan be used.
- rerun_dependent_tasksboolean
If true, repair all tasks that depend on the tasks in
rerun_tasks, even if they were previously successful. Can be also used in combination withrerun_all_failed_tasks.
- performance_targetstring
The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the performance target defined on the job level.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
Response
- repair_idint64
The ID of the repair. Must be provided in subsequent repairs using the
latest_repair_idfield to ensure sequential repairs.
Cancel Run GA
Cancel All Runs GA
POST
Cancels all active runs of a job. The runs are canceled asynchronously, so it doesn't prevent new runs from being started.
API scopes: jobs
Request body
- job_idint64
The canonical identifier of the job to cancel all runs of.
- all_queued_runsboolean
Optional boolean parameter to cancel all queued runs. If no job_id is provided, all queued runs in the workspace are canceled.
Get Run Output GA
GET
Retrieve the output and metadata of a single task run. When a notebook task returns
a value through the dbutils.notebook.exit() call, you can use this endpoint to retrieve
that value. <Databricks> restricts this API to returning the first 5 MB of the output.
To return a larger result, you can store job results in a cloud storage service.
This endpoint validates that the run_id parameter is valid and returns an HTTP status code 400 if the run_id parameter is invalid. Runs are automatically removed after 60 days. If you to want to reference them beyond 60 days, you must save old run results before they expire.
API scopes: jobs
Parameters
- run_idint64query
The canonical identifier for the run.
Response
- metadataobject
All details of the run except for its output.
Show child attributesHide child attributes
- job_idint64
The canonical identifier of the job that contains this run.
- run_idint64
The canonical identifier of the run. This ID is unique across all runs of all jobs.
- creator_user_namestring
The creator user name. This field won’t be included in the response if the user has already been deleted.
- number_in_jobint64
A unique identifier for this job run. This is set to the same value as
run_id.
- original_attempt_run_idint64
If this run is a retry of a prior run attempt, this field contains the run_id of the original attempt; otherwise, it is the same as the run_id.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- scheduleobject
The cron schedule that triggered this run if it was triggered by the periodic scheduler.
Show child attributesHide child attributes
- quartz_cron_expressionstring
A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
- timezone_idstring
A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
- pause_statusstring
Indicate whether this schedule is paused or not.
- cluster_specobject
A snapshot of the job’s cluster specification when this run was created.
Show child attributesHide child attributes
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- job_parametersarray of object
Job-level parameters used in the run
Show child attributesHide child attributes
- namestring
The name of the parameter
- defaultstring
The optional default value of the parameter
- valuestring
The value used in the run
- overriding_parametersobject
The parameters used for this run.
Show child attributesHide child attributes
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
Show child attributesHide child attributes
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- triggerstring
- trigger_infoobject
Show child attributesHide child attributes
- run_idint64
The run id of the Run Job task run
- run_namestring
An optional name for the run. The maximum length is 4096 bytes in UTF-8 encoding.
- run_page_urlstring
The URL to the detail page of the run.
- run_typestring
- tasksarray of object
The list of tasks performed by the run. Each task has its own
run_idwhich you can use to callJobsGetOutputto retrieve the run results. If more than 100 tasks are available, you can paginate through them usingjobs/getrun. Use thenext_page_tokenfield at the object root to determine if more results are available.Show child attributesHide child attributes
- run_idint64
The ID of the task run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- run_page_urlstring
- cluster_instanceobject
The cluster used for this run. If the run is specified to use a new cluster, this field is set once the Jobs service has requested a cluster for the run.
Show child attributesHide child attributes
- cluster_idstring
The canonical identifier for the cluster used by a run. This field is always available for runs on existing clusters. For runs on new clusters, it becomes available once the cluster is created. This value can be used to view logs by browsing to
/#setting/sparkui/$cluster_id/driver-logs. The logs continue to be available after the run completes.The response won’t include this field if the identifier is not available yet.
- spark_context_idstring
The canonical identifier for the Spark context used by a run. This field is filled in once the run begins execution. This value can be used to view the Spark UI by browsing to
/#setting/sparkui/$cluster_id/$spark_context_id. The Spark UI continues to be available after the run has completed.The response won’t include this field if the identifier is not available yet.
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks. If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task. Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
- sparse_checkoutobject
- resolved_valuesobject
Parameter values including resolved references
Show child attributesHide child attributes
- notebook_taskobjectRequired
- spark_jar_taskobjectRequired
- spark_python_taskobjectRequired
- spark_submit_taskobjectRequired
- python_wheel_taskobjectRequired
- dbt_taskobjectRequired
- sql_taskobjectRequired
- run_job_taskobjectRequired
- condition_taskobjectRequired
- simulation_taskobjectRequired
- pipeline_taskobjectRequiredBeta
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- task_keystring
A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
- descriptionstring
An optional description for this task.
- depends_onarray of object
An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete successfully before executing this task. The key is
task_key, and the value is the name assigned to the dependent task.Show child attributesHide child attributes
- task_keystring
The name of the task this task depends on.
- outcomestring
Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.
- run_ifstring
An optional value indicating the condition that determines whether the task should be run once its dependencies have been completed. When omitted, defaults to
ALL_SUCCESS. Seejobs/createfor a list of possible values.
- timeout_secondsint32
An optional timeout applied to each run of this job task. A value of
0means no timeout.
- email_notificationsobject
An optional set of email addresses notified when the task run begins or completes. The default behavior is to not send any emails.
Show child attributesHide child attributes
- on_startarray of string
A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_successarray of string
A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a
TERMINATEDlife_cycle_stateand aSUCCESSresult_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
- on_failurearray of string
A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an
INTERNAL_ERRORlife_cycle_stateor aFAILED, orTIMED_OUTresult_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
- on_duration_warning_threshold_exceededarray of string
A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. If no rule for theRUN_DURATION_SECONDSmetric is specified in thehealthfield for the job, notifications are not sent.
- on_streaming_backlog_exceededarray of stringPublic Preview
A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
- no_alert_for_skipped_runsboolean
If true, do not send email to recipients specified in
on_failureif the run is skipped. This field isdeprecated. Please use thenotification_settings.no_alert_for_skipped_runsfield.
- healthobject
Show child attributesHide child attributes
- rulesarray of object
- notification_settingsobject
Optional notification settings that are used when sending notifications to each of the
email_notificationsandwebhook_notificationsfor this task run.Show child attributesHide child attributes
- no_alert_for_skipped_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is skipped.
- no_alert_for_canceled_runsboolean
If true, do not send notifications to recipients specified in
on_failureif the run is canceled.
- alert_on_last_attemptboolean
If true, do not send notifications to recipients specified in
on_startfor the retried runs and do not send notifications to recipients specified inon_failureuntil the last retry of the run.
- webhook_notificationsobject
A collection of system notification IDs to notify when the run begins or completes. The default behavior is to not send any system notifications. Task webhooks respect the task notification settings.
Show child attributesHide child attributes
- on_startarray of object
An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the
on_startproperty.
- on_successarray of object
An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the
on_successproperty.
- on_failurearray of object
An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the
on_failureproperty.
- on_duration_warning_threshold_exceededarray of object
An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the
RUN_DURATION_SECONDSmetric in thehealthfield. A maximum of 3 destinations can be specified for theon_duration_warning_threshold_exceededproperty.
- on_streaming_backlog_exceededarray of objectPublic Preview
An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the
healthfield using the following metrics:STREAMING_BACKLOG_BYTES,STREAMING_BACKLOG_RECORDS,STREAMING_BACKLOG_SECONDS, orSTREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for theon_streaming_backlog_exceededproperty.
- environment_keystringRequired
The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
- disabledboolean
An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
- computeobjectBeta
Task level compute configuration.
Show child attributesHide child attributes
- hardware_acceleratorstringBeta
Hardware accelerator configuration for Serverless GPU workloads.
- notebook_taskobjectRequired
The task runs a notebook when the
notebook_taskfield is present.Show child attributesHide child attributes
- notebook_pathstring
The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
- base_parametersobject
Base parameters to be used for each run of this job. If the run is initiated by a call to
jobs/runNow with parameters specified, the two parameters maps are merged. If the same key is specified inbase_parametersand inrun-now, the value fromrun-nowis used. Use Task parameter variables to set parameters containing information about job runs.If the notebook takes a parameter that is not specified in the job’s
base_parametersor therun-nowoverride parameters, the default value from the notebook is used.Retrieve these parameters in a notebook using dbutils.widgets.get.
The JSON representation of this field cannot exceed 1MB.
- sourcestring
Optional location type of the notebook. When set to
WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set toGIT, the notebook will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Notebook is located in <Databricks> workspace.GIT: Notebook is located in cloud Git provider.
- warehouse_idstring
Optional
warehouse_idto run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.
- spark_jar_taskobjectRequired
The task runs a JAR when the
spark_jar_taskfield is present.Show child attributesHide child attributes
- jar_uristring
Deprecated since 04/2016. For classic compute, provide a
jarthrough thelibrariesfield instead. For serverless compute, provide ajarthough thejava_dependenciesfield inside theenvironmentslist.See the examples of classic and serverless compute usage at the top of the page.
- main_class_namestring
The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.
The code must use
SparkContext.getOrCreateto obtain a Spark context; otherwise, runs of the job fail.
- parametersarray of string
Parameters passed to the main method.
Use Task parameter variables to set parameters containing information about job runs.
- run_as_replboolean
Deprecated. A value of
falseis no longer supported.
- spark_python_taskobjectRequired
The task runs a Python file when the
spark_python_taskfield is present.Show child attributesHide child attributes
- python_filestring
The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with
/. For files stored in a remote repository, the path must be relative. This field is required.
- parametersarray of string
Command line parameters passed to the Python file.
Use Task parameter variables to set parameters containing information about job runs.
- sourcestring
Optional location type of the Python file. When set to
WORKSPACEor not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if thepython_filehas a URI format). When set toGIT, the Python file will be retrieved from a Git repository defined ingit_source.WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.GIT: The Python file is located in a remote Git repository.
- spark_submit_taskobjectRequired
(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.
Show child attributesHide child attributes
- parametersarray of string
Command-line parameters passed to spark submit.
Use Task parameter variables to set parameters containing information about job runs.
- pipeline_taskobjectRequired
The task triggers a pipeline update when the
pipeline_taskfield is present. Only pipelines configured to use triggered more are supported.Show child attributesHide child attributes
- pipeline_idstring
The full name of the pipeline task to execute.
- parametersobjectBeta
Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.
- full_refreshboolean
If true, triggers a full refresh on the spark declarative pipeline.
- refresh_selectionarray of stringBeta
A list of tables to update without fullRefresh.
- full_refresh_selectionarray of stringBeta
A list of tables to update with fullRefresh.
- reset_checkpoint_selectionarray of stringBeta
A list of streaming flows to reset checkpoints without clearing data.
- refresh_flow_selectionarray of stringBeta
Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.
- python_wheel_taskobjectRequired
The task runs a Python wheel when the
python_wheel_taskfield is present.Show child attributesHide child attributes
- package_namestring
Name of the package to execute
- entry_pointstring
Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using
$packageName.$entryPoint()
- parametersarray of string
Command-line parameters passed to Python wheel task. Leave it empty if
named_parametersis not null.
- named_parametersobject
Command-line parameters passed to Python wheel task in the form of
["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty ifparametersis not null.
- dbt_taskobjectRequired
The task runs one or more dbt commands when the
dbt_taskfield is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.Show child attributesHide child attributes
- project_directorystring
Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.
- commandsarray of string
A list of dbt commands to execute. All commands must start with
dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.
- schemastring
Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the
defaultschema is used.
- warehouse_idstring
ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the
--profiles-dircommand line argument.
- profiles_directorystring
Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.
- catalogstring
Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
- sourcestring
Optional location type of the project directory. When set to
WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set toGIT, the project will be retrieved from a Git repository defined ingit_source. If the value is empty, the task will useGITifgit_sourceis defined andWORKSPACEotherwise.WORKSPACE: Project is located in <Databricks> workspace.GIT: Project is located in cloud Git provider.
- sql_taskobjectRequired
The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the
sql_taskfield is present.Show child attributesHide child attributes
- parametersobject
Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.
- queryobjectRequired
If query, indicates that this job must execute a SQL query.
- dashboardobjectRequired
If dashboard, indicates that this job must refresh a SQL dashboard.
- alertobjectRequired
If alert, indicates that this job must refresh a SQL alert.
- fileobjectRequired
If file, indicates that this job runs a SQL file in a remote Git repository.
- warehouse_idstring
The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.
- run_job_taskobjectRequired
The task triggers another job when the
run_job_taskfield is present.Show child attributesHide child attributes
- job_idint64
ID of the job to trigger.
- job_parametersobject
Job-level parameters used to trigger the job.
- pipeline_paramsobject
Controls whether the pipeline should perform a full refresh
- condition_taskobjectRequired
The task evaluates a condition that can be used to control the execution of other tasks when the
condition_taskfield is present. The condition task does not require a cluster to execute and does not support retries or notifications.Show child attributesHide child attributes
- opstring
EQUAL_TO,NOT_EQUALoperators perform string comparison of their operands. This means that“12.0” == “12”will evaluate tofalse.GREATER_THAN,GREATER_THAN_OR_EQUAL,LESS_THAN,LESS_THAN_OR_EQUALoperators perform numeric comparison of their operands.“12.0” >= “12”will evaluate totrue,“10.0” >= “12”will evaluate tofalse.
The boolean comparison to task values can be implemented with operators
EQUAL_TO,NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to“true”or“false”for the comparison.
- leftstring
The left operand of the condition task. Can be either a string value or a job state or parameter reference.
- rightstring
The right operand of the condition task. Can be either a string value or a job state or parameter reference.
- outcomestring
The condition expression evaluation result. Filled in if the task was successfully completed. Can be
"true"or"false"
- for_each_taskobjectRequired
The task executes a nested task for every input provided when the
for_each_taskfield is present.Show child attributesHide child attributes
- inputsstring
Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
- concurrencyint32
An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
- taskobject
Configuration for the task that will be run for each element in the array
- clean_rooms_notebook_taskobjectRequired
The task runs a clean rooms notebook when the
clean_rooms_notebook_taskfield is present.Show child attributesHide child attributes
- clean_room_namestring
The clean room that the notebook belongs to.
- notebook_namestring
Name of the notebook being run.
- etagstring
Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the
cleanroomassets/getAPI.
- notebook_base_parametersobject
Base parameters to be used for the clean room notebook job.
- alert_taskobjectRequiredPublic Preview
The task evaluates a <Databricks> alert and sends notifications to subscribers when the
alert_taskfield is present.Show child attributesHide child attributes
- alert_idstringPublic Preview
The alert_id is the canonical identifier of the alert.
- warehouse_idstringPublic Preview
The warehouse_id identifies the warehouse settings used by the alert task.
- workspace_pathstringPublic Preview
The workspace_path is the path to the alert file in the workspace. The path:
- must start with "/Workspace"
- must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.
- subscribersarray of objectPublic Preview
The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.
- power_bi_taskobjectRequiredPublic Preview
The task triggers a Power BI semantic model update when the
power_bi_taskfield is present.Show child attributesHide child attributes
- tablesarray of objectPublic Preview
The tables to be exported to Power BI
- warehouse_idstringPublic Preview
The SQL warehouse ID to use as the Power BI data source
- power_bi_modelobjectPublic Preview
The semantic model to update
- connection_resource_namestringPublic Preview
The resource name of the UC connection to authenticate from <Databricks> to Power BI
- refresh_after_updatebooleanPublic Preview
Whether the model should be refreshed after the update
- dashboard_taskobjectRequired
The task refreshes a dashboard and sends a snapshot to subscribers.
Show child attributesHide child attributes
- subscriptionobject
Optional: subscription configuration for sending the dashboard snapshot.
- warehouse_idstring
Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
- dashboard_idstring
The identifier of the dashboard to refresh.
- ai_runtime_taskobjectRequiredPublic Preview
The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.
Show child attributesHide child attributes
- experimentstringPublic Preview
MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set
mlflow_experiment_directory.
- deploymentsarray of objectPublic Preview
Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.
- mlflow_runstringPublic Preview
Optional display name for the MLflow run created under
experiment. If omitted, MLflow generates a default name.
- mlflow_experiment_directorystringPublic Preview
Optional workspace directory under which the MLflow experiment named in
experimentis created. Must start with/Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
- docker_image_urlstringBeta
Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format:
{organization}/{repository}:{tag}
- existing_cluster_idstringRequired
If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
- new_clusterobjectRequired
If new_cluster, a description of a new cluster that is created for each run.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- job_cluster_keystringRequired
If job_cluster_key, this task is executed reusing the cluster specified in
job.settings.job_clusters.
- librariesarray of object
An optional list of libraries to be installed on the cluster. The default value is an empty list.
Show child attributesHide child attributes
- jarstringRequired
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "jar": "/Workspace/path/to/library.jar" },{ "jar" : "/Volumes/path/to/library.jar" }or{ "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- eggstringRequired
Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.
- pypiobjectRequired
Specification of a PyPi library to be installed. For example:
{ "package": "simplejson" }
- mavenobjectRequired
Specification of a maven library to be installed. For example:
{ "coordinates": "org.jsoup:jsoup:1.7.2" }
- cranobjectRequired
Specification of a CRAN library to be installed as part of the library
- whlstringRequired
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.AzureURI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.GCPURI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example:
{ "whl": "/Workspace/path/to/library.whl" },{ "whl" : "/Volumes/path/to/library.whl" }or{ "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.
- requirementsstringRequired
URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example:
{ "requirements": "/Workspace/path/to/requirements.txt" }or{ "requirements" : "/Volumes/path/to/requirements.txt" }
- max_retriesint32
An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the
FAILEDresult_state orINTERNAL_ERRORlife_cycle_state. The value-1means to retry indefinitely and the value0means to never retry.
- min_retry_interval_millisint32
An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
- retry_on_timeoutboolean
An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
- disable_auto_optimizationboolean
An option to disable auto optimization in serverless
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- descriptionstring
Description of the run
- attempt_numberint32
The sequence number of this run attempt for a triggered job run. The initial attempt of a run has an attempt_number of 0. If the initial run attempt fails, and the job has a retry policy (
max_retries> 0), subsequent runs are created with anoriginal_attempt_run_idof the original attempt’s ID and an incrementingattempt_number. Runs are retried only until they succeed, and the maximumattempt_numberis the same as themax_retriesvalue for the job.
- job_clustersarray of object
A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings. If more than 100 job clusters are available, you can paginate through them using
jobs/getrun.Show child attributesHide child attributes
- job_cluster_keystring
A unique name for the job cluster. This field is required and must be unique within the job.
JobTaskSettingsmay refer to this field to determine which cluster to launch for the task execution.
- new_clusterobject
If new_cluster, a description of a cluster that is created for each task.
Show child attributesHide child attributes
- apply_policy_default_valuesboolean
- cluster_namestring
Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.
- spark_versionstring
The Spark version of the cluster, e.g.
3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.
- spark_confobject
An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptionsandspark.executor.extraJavaOptionsrespectively.
- aws_attributesobject
Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.
- azure_attributesobject
Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.
- gcp_attributesobject
Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.
- node_type_idstring
This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.
- driver_node_type_idstring
The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as
node_type_iddefined above.This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.
- worker_node_type_flexibilityobject
Flexible node type configuration for worker nodes.
- driver_node_type_flexibilityobject
Flexible node type configuration for the driver node.
- ssh_public_keysarray of string
SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name
ubuntuon port2200. Up to 10 keys can be specified.
- custom_tagsobject
Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to
default_tags. Notes:-
Currently, <Databricks> allows at most 45 custom tags
-
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags
-
- cluster_log_confobject
The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is$destination/$clusterId/driver, while the destination of executor logs is$destination/$clusterId/executor.
- spark_env_varsobject
An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e.,
export X='Y') while launching the driver and workers.In order to specify an additional set of
SPARK_DAEMON_JAVA_OPTS, we recommend appending them to$SPARK_DAEMON_JAVA_OPTSas shown in the example below. This ensures that all default databricks managed environmental variables are included as well.Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"}or{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
- autotermination_minutesint32
Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.
- enable_elastic_diskboolean
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWSAutoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
- init_scriptsarray of object
The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If
cluster_log_confis specified, init script logs are sent to<destination>/<cluster-ID>/init_scripts.
- docker_imageobject
Custom docker image BYOC
- instance_pool_idstring
The optional ID of the instance pool to which the cluster belongs.
- single_user_namestring
Single user name if data_security_mode is
SINGLE_USER
- policy_idstring
The ID of the cluster policy used to create the cluster if applicable.
- enable_local_disk_encryptionboolean
Whether to enable LUKS on cluster VMs' local disks
- driver_instance_pool_idstring
The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.
- workload_typeobject
- data_security_modestring
- runtime_enginestring
Determines the cluster's runtime engine, either standard or Photon.
This field is not compatible with legacy
spark_versionvalues that contain-photon-. Remove-photon-from thespark_versionand setruntime_enginetoPHOTON.If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.
- kindstring
- use_ml_runtimeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.effective_spark_versionis determined byspark_version(DBR release), this fielduse_ml_runtime, and whethernode_type_idis gpu node or not.
- is_single_nodeboolean
This field can only be used when
kind = CLASSIC_PREVIEW.When set to true, <Databricks> will automatically set single node related
custom_tags,spark_conf, andnum_workers
- remote_disk_throughputint32
If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.
- total_initial_remote_disk_sizeint32
If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.
- dependency_modestringBeta
Controls dependency configuration for the cluster.
- num_workersint32Required
Number of worker nodes that this cluster should have. A cluster has one Spark Driver and
num_workersExecutors for a total ofnum_workers+ 1 Spark nodes.Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in
spark_infowill gradually increase from 5 to 10 as the new nodes are provisioned.
- autoscaleobjectRequired
Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.
- git_sourceobject
An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.
If
git_sourceis set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by settingsourcetoWORKSPACEon the task.Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used,
git_sourcemust be defined on the job.Show child attributesHide child attributes
- git_urlstring
URL of the repository to be cloned by this job.
- git_providerstring
Unique identifier of the service used to host the Git repository. The value is case insensitive.
- git_branchstringRequired
Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
- git_tagstringRequired
Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
- git_commitstringRequired
Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
- git_snapshotobject
Show child attributesHide child attributes
- used_commitstring
Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
- sparse_checkoutobject
Show child attributesHide child attributes
- patternsarray of string
List of patterns to include for sparse checkout.
- repair_historyarray of object
The repair history of the run.
Show child attributesHide child attributes
- typestring
The repair history item type. Indicates whether a run is the original run or a repair run.
- start_timeint64
The start time of the (repaired) run.
- end_timeint64
The end time of the (repaired) run.
- stateobject
Deprecated. Please use the
statusfield instead.Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- state_messagestring
A descriptive message for the current state. This field is unstructured, and its exact format is subject to change.
- user_cancelled_or_timedoutboolean
A value indicating whether a run was canceled manually by a user or by the scheduler because the run timed out.
- queue_reasonstring
The reason indicating why the run was queued.
- idint64
The ID of the repair. Only returned for the items that represent a repair in
repair_history.
- task_run_idsarray of int64
The run IDs of the task runs that ran as part of this repair history item.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- statusobject
Show child attributesHide child attributes
- statestring
- termination_detailsobject
If the run is in a TERMINATING or TERMINATED state, details about the reason for terminating the run.
Show child attributesHide child attributes
- codestring
- typestring
- messagestring
A descriptive message with the termination details. This field is unstructured and the format might change.
- queue_detailsobject
If the run was queued, details about the reason for queuing the run.
Show child attributesHide child attributes
- codestring
- messagestring
A descriptive message with the queuing details. This field is unstructured, and its exact format is subject to change.
- job_run_idint64
ID of the job run that this run belongs to. For legacy and single-task job runs the field is populated with the job run ID. For task runs, the field is populated with the ID of the job run that the task run belongs to.
- has_moreboolean
Indicates if the run has more array properties (
tasks,job_clusters) that are not shown. They can be accessed viajobs/getrunendpoint. It is only relevant for API 2.2jobs/listrunsrequests withexpand_tasks=true.
- effective_performance_targetstring
The actual performance target used by the serverless run during execution. This can differ from the client-set performance target on the request depending on whether the performance mode is supported by the job type.
STANDARD: Enables cost-efficient execution of serverless workloads.PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.
- start_timeint64
The time at which this run was started in epoch milliseconds (milliseconds since 1/1/1970 UTC). This may not be the time when the job task starts executing, for example, if the job is scheduled to run on a new cluster, this is the time the cluster creation call is issued.
- setup_durationint64
The time in milliseconds it took to set up the cluster. For runs that run on new clusters this is the cluster creation time, for runs that run on existing clusters this time should be very short. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thesetup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- execution_durationint64
The time in milliseconds it took to execute the commands in the JAR or notebook until they completed, failed, timed out, were cancelled, or encountered an unexpected error. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Theexecution_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- cleanup_durationint64
The time in milliseconds it took to terminate the cluster and clean up any associated artifacts. The duration of a task run is the sum of the
setup_duration,execution_duration, and thecleanup_duration. Thecleanup_durationfield is set to 0 for multitask job runs. The total duration of a multitask job run is the value of therun_durationfield.
- end_timeint64
The time at which this run ended in epoch milliseconds (milliseconds since 1/1/1970 UTC). This field is set to 0 if the job is still running.
- run_durationint64
The time in milliseconds it took the job run and all of its repairs to finish.
- queue_durationint64
The time in milliseconds that the run has spent in the queue.
- errorstring
An error message indicating why a task failed or why output is not available. The message is unstructured, and its exact format is subject to change.
- infostring
- notebook_outputobjectRequired
The output of a notebook task, if available. A notebook task that terminates (either successfully or with a failure) without calling
dbutils.notebook.exit()is considered to have an empty output. This field is set but its result value is empty. <Databricks> restricts this API to return the first 5 MB of the output. To return a larger result, use the ClusterLogConf field to configure log storage for the job cluster.Show child attributesHide child attributes
- resultstring
The value passed to dbutils.notebook.exit(). <Databricks> restricts this API to return the first 5 MB of the value. For a larger result, your job can store the results in a cloud storage service. This field is absent if
dbutils.notebook.exit()was never called.
- truncatedboolean
Whether or not the result was truncated.
- sql_outputobjectRequired
The output of a SQL task, if available.
Show child attributesHide child attributes
- query_outputobjectRequired
The output of a SQL query task, if available.
Show child attributesHide child attributes
- query_textstring
The text of the SQL query. Can Run permission of the SQL query is required to view this field.
- endpoint_idstring
- sql_statementsarray of object
Information about SQL statements executed in the run.
Show child attributesHide child attributes
- lookup_keystring
A key that can be used to look up query details.
- output_linkstring
The link to find the output results.
- warehouse_idstring
The canonical identifier of the SQL warehouse.
- dashboard_outputobjectRequired
The output of a SQL dashboard task, if available.
Show child attributesHide child attributes
- widgetsarray of object
Widgets executed in the run. Only SQL query based widgets are listed.
Show child attributesHide child attributes
- widget_idstring
The canonical identifier of the SQL widget.
- widget_titlestring
The title of the SQL widget.
- output_linkstring
The link to find the output results.
- statusstring
The execution status of the SQL widget.
- errorobject
The information about the error when execution fails.
- start_timeint64
Time (in epoch milliseconds) when execution of the SQL widget starts.
- end_timeint64
Time (in epoch milliseconds) when execution of the SQL widget ends.
- warehouse_idstring
The canonical identifier of the SQL warehouse.
- alert_outputobjectRequired
The output of a SQL alert task, if available.
Show child attributesHide child attributes
- query_textstring
The text of the SQL query. Can Run permission of the SQL query associated with the SQL alert is required to view this field.
- sql_statementsarray of object
Information about SQL statements executed in the run.
Show child attributesHide child attributes
- lookup_keystring
A key that can be used to look up query details.
- output_linkstring
The link to find the output results.
- warehouse_idstring
The canonical identifier of the SQL warehouse.
- alert_statestring
- dbt_outputobjectRequired
The output of a dbt task, if available.
Show child attributesHide child attributes
- artifacts_linkstring
A pre-signed URL to download the (compressed) dbt artifacts. This link is valid for a limited time (30 minutes). This information is only available after the run has finished.
- artifacts_headersobject
An optional map of headers to send when retrieving the artifact from the
artifacts_link.
- run_job_outputobjectRequired
The output of a run job task, if available
Show child attributesHide child attributes
- run_idint64
The run id of the triggered job run
- clean_rooms_notebook_outputobjectRequired
The output of a clean rooms notebook task, if available
Show child attributesHide child attributes
- clean_room_job_run_stateobject
The run state of the clean rooms notebook task.
Show child attributesHide child attributes
- life_cycle_statestring
A value indicating the run's current lifecycle state. This field is always available in the response. Note: Additional states might be introduced in future releases.
- result_statestring
A value indicating the run's result. This field is only available for terminal lifecycle states. Note: Additional states might be introduced in future releases.
- notebook_outputobject
The notebook output for the clean room run
Show child attributesHide child attributes
- resultstring
The value passed to dbutils.notebook.exit(). <Databricks> restricts this API to return the first 5 MB of the value. For a larger result, your job can store the results in a cloud storage service. This field is absent if
dbutils.notebook.exit()was never called.
- truncatedboolean
Whether or not the result was truncated.
- output_schema_infoobject
Information on how to access the output schema for the clean room run
Show child attributesHide child attributes
- catalog_namestring
- schema_namestring
- expiration_timeint64
The expiration time for the output schema as a Unix timestamp in milliseconds.
- dashboard_outputobjectRequired
The output of a dashboard task, if available
Show child attributesHide child attributes
- page_snapshotsarray of object
Should only be populated for manual PDF download jobs.
Show child attributesHide child attributes
- page_display_namestring
- widget_error_detailsarray of object
Show child attributesHide child attributes
- messagestring
- alert_outputobjectRequiredPublic Preview
The output of an alert task, if available
Show child attributesHide child attributes
- alert_statestringPublic Preview
- logsstring
The output from tasks that write to standard streams (stdout/stderr) such as spark_jar_task, spark_python_task, python_wheel_task.
It's not supported for the notebook_task, pipeline_task or spark_submit_task.
<Databricks> restricts this API to return the last 5 MB of these logs.
- logs_truncatedboolean
Whether the logs are truncated.
- error_tracestring
If there was an error executing the run, this field contains any available stack traces.
Export Run GA
GET
Export and retrieve the job run task.
API scopes: jobs
Parameters
- run_idint64query
The canonical identifier for the run. This field is required.
- views_to_exportstringquery
Which views to export (CODE, DASHBOARDS, or ALL). Defaults to CODE.
Response
- viewsarray of object
The exported content in HTML format (one for every view item). To extract the HTML notebook from the JSON response, download and run this Python script.
Show child attributesHide child attributes
- contentstring
Content of the view.
- namestring
Name of the view item. In the case of code view, it would be the notebook’s name. In the case of dashboard view, it would be the dashboard’s name.
- typestring
Type of the view item.