Job

Get GA

GET /api/2.2/jobs/get

Retrieves the details for a single job.

Large arrays in the results will be paginated when they exceed 100 elements. A request for a single job will return all properties for that job, and the first 100 elements of array properties (tasks, job_clusters, environments and parameters). Use the next_page_token field to check for more results and pass its value as the page_token in subsequent requests. If any array properties have more than 100 elements, additional results will be returned on subsequent requests. Arrays without additional results will be empty on later pages.

API scopes: jobs

Parameters

job_idint64query: The canonical identifier of the job to retrieve information about. This field is required.
Example: 11223344

include_trigger_statebooleanquery: Flag that indicates that trigger state should be included in the response.
Default: false

page_tokenstringquery: Use next_page_token returned from the previous GetJob response to request the next page of the job's array properties.
Example: CAAos-uriYcxMN7_rt_v7B4=

Response

next_page_tokenstring: A token that can be used to list the next page of array properties.
Example: CAAos-uriYcxMN7_rt_v7B4=

job_idint64: The canonical identifier for this job.
Example: 11223344

creator_user_namestring: The creator user name. This field won’t be included in the response if the user has already been deleted.
Example: user.name@databricks.com

run_as_user_namestring

The email of an active workspace user or the application ID of a service principal that the job runs as. This value can be changed by setting the run_as field when creating or updating a job.

By default, run_as_user_name is based on the current job settings and is set to the creator of the job if job access control is disabled or to the user with the is_owner permission if job access control is enabled.

Example: user.name@databricks.com

settingsobject

Settings for this job and all of its runs. These settings can be updated using the resetJob method.

Show child attributesHide child attributes

namestring: An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.
Default: Untitled
Example: A multitask job
Constraints: <= 4096 characters

descriptionstring: An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.
Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.
Constraints: <= 27700 characters

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes

on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes

idstring

on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes

idstring

on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes

idstring

on_duration_warning_threshold_exceededarray of object

An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

Show child attributesHide child attributes

idstring

on_streaming_backlog_exceededarray of objectPublic Preview

An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

Show child attributesHide child attributes

idstring

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

timeout_secondsint32: An optional timeout applied to each run of this job. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

Show child attributesHide child attributes

metricstring: Values: RUN_DURATION_SECONDS, STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, STREAMING_BACKLOG_FILES
Example: RUN_DURATION_SECONDS

opstring: Values: GREATER_THAN
Example: GREATER_THAN

valueint64: Specifies the threshold value that the health metric should obey to satisfy the health rule.
Example: 10

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

quartz_cron_expressionstring: A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
Example: 20 30 * * * ?

timezone_idstring: A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
Example: Europe/London

pause_statusstring: Indicate whether this schedule is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

triggerobject

A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

pause_statusstring: Whether this trigger is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

file_arrivalobjectRequired

File arrival trigger settings.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

periodicobjectRequired

Periodic trigger settings.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

table_updateobjectRequired

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes

pause_statusstring: Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

task_retry_modestring: Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.
Values: NEVER, ON_FAILURE

max_concurrent_runsint32: An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.
Default: 1
Example: 10

tasksarray of object

A list of task specifications to be executed by this job. It supports up to 1000 elements in write endpoints (jobs/create, jobs/reset, jobs/update, jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using jobs/get. Use the next_page_token field at the object root to determine if more results are available.

Show child attributesHide child attributes

task_keystring: A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

depends_onarray of object

An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the run_if condition is true. The key is task_key, and the value is the name assigned to the dependent task.

Show child attributesHide child attributes

task_keystring: The name of the task this task depends on.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

outcomestring: Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

ALL_SUCCESS: All dependencies have executed and succeeded
AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
NONE_FAILED: None of the dependencies have failed and at least one was executed
ALL_DONE: All dependencies have been completed
AT_LEAST_ONE_FAILED: At least one dependency failed
ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values: ALL_SUCCESS, ALL_DONE, NONE_FAILED, AT_LEAST_ONE_SUCCESS, ALL_FAILED, AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32: An optional timeout applied to each run of this job task. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

email_notificationsobject

An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

Show child attributesHide child attributes

on_startarray of object: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

on_successarray of object: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

on_failurearray of object: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

on_duration_warning_threshold_exceededarray of object: An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

on_streaming_backlog_exceededarray of objectPublic Preview: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

descriptionstring: An optional description for this task.
Example: This is the description for this task.
Constraints: <= 1000 characters

environment_keystringRequired: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

disabledboolean: An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
Default: false

computeobjectBeta

Task level compute configuration.

Show child attributesHide child attributes

hardware_acceleratorstringBeta: Hardware accelerator configuration for Serverless GPU workloads.
Values: GPU_1X_A10, GPU_8X_H100

notebook_taskobjectRequired

The task runs a notebook when the notebook_task field is present.

Show child attributesHide child attributes

notebook_pathstring: The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
Example: /Users/user.name@databricks.com/notebook_to_run

base_parametersobject

Base parameters to be used for each run of this job. If the run is initiated by a call to jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is specified in base_parameters and in run-now, the value from run-now is used. Use Task parameter variables to set parameters containing information about job runs.

If the notebook takes a parameter that is not specified in the job’s base_parameters or the run-now override parameters, the default value from the notebook is used.

Retrieve these parameters in a notebook using dbutils.widgets.get.

The JSON representation of this field cannot exceed 1MB.

sourcestring

Optional location type of the notebook. When set to WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set to GIT, the notebook will be retrieved from a Git repository defined in git_source. If the value is empty, the task will use GIT if git_source is defined and WORKSPACE otherwise.

WORKSPACE: Notebook is located in <Databricks> workspace.
GIT: Notebook is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

warehouse_idstring

Optional warehouse_id to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.

Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.

Example: ab12cd34efgh567i

spark_jar_taskobjectRequired

The task runs a JAR when the spark_jar_task field is present.

Show child attributesHide child attributes

jar_uristring

Deprecated since 04/2016. For classic compute, provide a jar through the libraries field instead. For serverless compute, provide a jar though the java_dependencies field inside the environments list.

See the examples of classic and serverless compute usage at the top of the page.

main_class_namestring

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.

The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

Example: com.databricks.ComputeModels

parametersarray of string

Parameters passed to the main method.

Use Task parameter variables to set parameters containing information about job runs.

run_as_replboolean: Deprecated. A value of false is no longer supported.
Default: true

spark_python_taskobjectRequired

The task runs a Python file when the spark_python_task field is present.

Show child attributesHide child attributes

python_filestring: The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with /. For files stored in a remote repository, the path must be relative. This field is required.
Example: dbfs:/path/to/file.py

parametersarray of string

Command line parameters passed to the Python file.

Use Task parameter variables to set parameters containing information about job runs.

sourcestring

Optional location type of the Python file. When set to WORKSPACE or not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if the python_file has a URI format). When set to GIT, the Python file will be retrieved from a Git repository defined in git_source.

WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.
GIT: The Python file is located in a remote Git repository.

Values: WORKSPACE, GIT

Example: WORKSPACE

spark_submit_taskobjectRequired

(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

Show child attributesHide child attributes

parametersarray of string

Command-line parameters passed to spark submit.

Use Task parameter variables to set parameters containing information about job runs.

pipeline_taskobjectRequired

The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

Show child attributesHide child attributes

pipeline_idstring: The full name of the pipeline task to execute.
Example: a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5

parametersobjectBeta: Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

python_wheel_taskobjectRequired

The task runs a Python wheel when the python_wheel_task field is present.

Show child attributesHide child attributes

package_namestring: Name of the package to execute

entry_pointstring: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using $packageName.$entryPoint()

parametersarray of string: Command-line parameters passed to Python wheel task. Leave it empty if named_parameters is not null.

named_parametersobject: Command-line parameters passed to Python wheel task in the form of ["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty if parameters is not null.

dbt_taskobjectRequired

The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

Show child attributesHide child attributes

project_directorystring: Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.

commandsarray of string: A list of dbt commands to execute. All commands must start with dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.

schemastring: Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the default schema is used.

warehouse_idstring: ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the --profiles-dir command line argument.
Example: 30dade0507d960d1

profiles_directorystring: Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.

catalogstring: Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
Example: main

sourcestring

Optional location type of the project directory. When set to WORKSPACE, the project will be retrieved from the local <Databricks> workspace. When set to GIT, the project will be retrieved from a Git repository defined in git_source. If the value is empty, the task will use GIT if git_source is defined and WORKSPACE otherwise.

WORKSPACE: Project is located in <Databricks> workspace.
GIT: Project is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

sql_taskobjectRequired

The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the sql_task field is present.

Show child attributesHide child attributes

parametersobject: Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.

queryobjectRequired: If query, indicates that this job must execute a SQL query.

dashboardobjectRequired: If dashboard, indicates that this job must refresh a SQL dashboard.

alertobjectRequired: If alert, indicates that this job must refresh a SQL alert.

fileobjectRequired: If file, indicates that this job runs a SQL file in a remote Git repository.

warehouse_idstring: The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.

run_job_taskobjectRequired

The task triggers another job when the run_job_task field is present.

Show child attributesHide child attributes

job_idint64: ID of the job to trigger.

job_parametersobject: Job-level parameters used to trigger the job.

pipeline_paramsobject: Controls whether the pipeline should perform a full refresh

condition_taskobjectRequired

The task evaluates a condition that can be used to control the execution of other tasks when the condition_task field is present. The condition task does not require a cluster to execute and does not support retries or notifications.

Show child attributesHide child attributes

opstring

EQUAL_TO, NOT_EQUAL operators perform string comparison of their operands. This means that “12.0” == “12” will evaluate to false.
GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL operators perform numeric comparison of their operands. “12.0” >= “12” will evaluate to true, “10.0” >= “12” will evaluate to false.

The boolean comparison to task values can be implemented with operators EQUAL_TO, NOT_EQUAL. If a task value was set to a boolean value, it will be serialized to “true” or “false” for the comparison.

Values: EQUAL_TO, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL, NOT_EQUAL

leftstring: The left operand of the condition task. Can be either a string value or a job state or parameter reference.

rightstring: The right operand of the condition task. Can be either a string value or a job state or parameter reference.

outcomestring: The condition expression evaluation result. Filled in if the task was successfully completed. Can be "true" or "false"

for_each_taskobjectRequired

The task executes a nested task for every input provided when the for_each_task field is present.

Show child attributesHide child attributes

inputsstring: Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
Constraints: <= 5000 characters

concurrencyint32: An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
Example: 20
Constraints: [ 1 .. 100 ]

taskobject: Configuration for the task that will be run for each element in the array

clean_rooms_notebook_taskobjectRequired

The task runs a clean rooms notebook when the clean_rooms_notebook_task field is present.

Show child attributesHide child attributes

clean_room_namestring: The clean room that the notebook belongs to.

notebook_namestring: Name of the notebook being run.

etagstring: Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the cleanroomassets/get API.

notebook_base_parametersobject: Base parameters to be used for the clean room notebook job.

alert_taskobjectRequiredPublic Preview

The task evaluates a <Databricks> alert and sends notifications to subscribers when the alert_task field is present.

Show child attributesHide child attributes

alert_idstringPublic Preview: The alert_id is the canonical identifier of the alert.

warehouse_idstringPublic Preview: The warehouse_id identifies the warehouse settings used by the alert task.

workspace_pathstringPublic Preview

The workspace_path is the path to the alert file in the workspace. The path:

must start with "/Workspace"
must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.

Example: /Workspace/Users/user@company.com/example.dbalert.json

subscribersarray of objectPublic Preview: The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.

power_bi_taskobjectRequiredPublic Preview

The task triggers a Power BI semantic model update when the power_bi_task field is present.

Show child attributesHide child attributes

tablesarray of objectPublic Preview: The tables to be exported to Power BI

warehouse_idstringPublic Preview: The SQL warehouse ID to use as the Power BI data source

power_bi_modelobjectPublic Preview: The semantic model to update

connection_resource_namestringPublic Preview: The resource name of the UC connection to authenticate from <Databricks> to Power BI

refresh_after_updatebooleanPublic Preview: Whether the model should be refreshed after the update

dashboard_taskobjectRequired

The task refreshes a dashboard and sends a snapshot to subscribers.

Show child attributesHide child attributes

subscriptionobject: Optional: subscription configuration for sending the dashboard snapshot.

warehouse_idstring: Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
Example: 47bb1c472649e711

dashboard_idstring: The identifier of the dashboard to refresh.
Example: 01ef0cb45e2a1da4a61950e9b8789ce9

ai_runtime_taskobjectRequiredPublic Preview

The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.

Show child attributesHide child attributes

experimentstringPublic Preview: MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set mlflow_experiment_directory.
Example: llama-fine-tune

deploymentsarray of objectPublic Preview: Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.

mlflow_runstringPublic Preview: Optional display name for the MLflow run created under experiment. If omitted, MLflow generates a default name.
Example: llama-fine-tune-lora-r16

mlflow_experiment_directorystringPublic Preview: Optional workspace directory under which the MLflow experiment named in experiment is created. Must start with /Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
Example: /Workspace/Users/alice@databricks.com/experiments

docker_image_urlstringBeta: Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format: {organization}/{repository}:{tag}
Example: nvidia/cuda:13.3.0-cudnn-devel-ubuntu24.04

existing_cluster_idstringRequired: If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
Example: 0923-164208-meows279

new_clusterobjectRequired

If new_cluster, a description of a new cluster that is created for each run.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

This field, along with node_type_id, should not be set if virtual_cluster_size is set. If both driver_node_type_id, node_type_id, and virtual_cluster_size are specified, driver_node_type_id and node_type_id take precedence.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

job_cluster_keystringRequired: If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

librariesarray of object

An optional list of libraries to be installed on the cluster. The default value is an empty list.

Show child attributesHide child attributes

jarstringRequired: URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

eggstringRequired: Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.

pypiobjectRequired: Specification of a PyPi library to be installed. For example: { "package": "simplejson" }

mavenobjectRequired: Specification of a maven library to be installed. For example: { "coordinates": "org.jsoup:jsoup:1.7.2" }

cranobjectRequired: Specification of a CRAN library to be installed as part of the library

whlstringRequired: URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

requirementsstringRequired: URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example: { "requirements": "/Workspace/path/to/requirements.txt" } or { "requirements" : "/Volumes/path/to/requirements.txt" }

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

job_clustersarray of object

A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings.

Constraints: <= 100 items

Show child attributesHide child attributes

job_cluster_keystring: A unique name for the job cluster. This field is required and must be unique within the job. JobTaskSettings may refer to this field to determine which cluster to launch for the task execution.
Example: auto_scaling_cluster
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

new_clusterobject

If new_cluster, a description of a cluster that is created for each task.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

git_sourceobject

An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.

If git_source is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by setting source to WORKSPACE on the task.

Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used, git_source must be defined on the job.

Show child attributesHide child attributes

git_urlstring: URL of the repository to be cloned by this job.
Example: https://github.com/databricks/databricks-cli
Constraints: <= 300 characters

git_providerstring: Unique identifier of the service used to host the Git repository. The value is case insensitive.

git_branchstringRequired: Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
Example: main
Constraints: <= 255 characters

git_tagstringRequired: Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
Example: release-1.0.0
Constraints: <= 255 characters

git_commitstringRequired: Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
Example: e0056d01
Constraints: <= 64 characters

git_snapshotobject

Show child attributesHide child attributes

used_commitstring: Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
Example: 4506fdf41e9fa98090570a34df7a5bce163ff15f

sparse_checkoutobject

Show child attributesHide child attributes

patternsarray of string: List of patterns to include for sparse checkout.

tagsobject: A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added to the job.

formatstring: Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is always set to "MULTI_TASK".
Values: SINGLE_TASK, MULTI_TASK
Example: MULTI_TASK

queueobject

The queue settings of the job.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

parametersarray of object

Job-level parameter definitions

Show child attributesHide child attributes

namestring: The name of the defined parameter. May only contain alphanumeric characters, _, -, and .
Example: table
Constraints: ^[\\w\\-.]+$

defaultstring: Default value of the parameter.
Example: users

run_asobject

The user or service principal that the job runs as, if specified in the request. This field indicates the explicit configuration of run_as for the job. To find the value in all cases, explicit or implicit, use run_as_user_name.

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Non-admin users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

edit_modestring

Edit mode of the job.

UI_LOCKED: The job is in a locked UI state and cannot be modified.
EDITABLE: The job is in an editable state and can be modified.

Values: UI_LOCKED, EDITABLE

deploymentobject

Deployment information for jobs managed by external sources.

Show child attributesHide child attributes

kindstring

The kind of deployment that manages the job.

BUNDLE: The job is managed by Databricks Asset Bundle.
SYSTEM_MANAGED: The job is managed by <Databricks> and is read-only.

Values: BUNDLE, SYSTEM_MANAGED

metadata_file_pathstring: Path of the file that contains deployment metadata.

environmentsarray of object

A list of task execution environment specifications that can be referenced by serverless tasks of this job. For serverless notebook tasks, if the environment_key is not specified, the notebook environment will be used if present. If a jobs environment is specified, it will override the notebook environment. For other serverless tasks, the task environment is required to be specified using environment_key in the task settings.

Constraints: <= 10 items

Show child attributesHide child attributes

environment_keystring: The key of an environment. It has to be unique within a job.

specobject

Show child attributesHide child attributes

clientstring: Use environment_version instead.
Example: 1

dependenciesarray of string: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a valid pip requirements file line per https://pip.pypa.io/en/stable/reference/requirements-file-format/. Allowed dependencies include a requirement specifier, an archive URL, a local project path (such as WSFS or UC Volumes in <Databricks>), or a VCS project URL.

base_environmentstring: The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
AWS
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see https://docs.databricks.com/aws/en/compute/serverless/dependencies#base-environments-for-job-tasks.
Azure
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://learn.microsoft.com/azure/databricks/compute/serverless/dependencies#base-environments-for-job-tasks.
GCP
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://docs.databricks.com/gcp/en/compute/serverless/dependencies#base-environments-for-job-tasks.

environment_versionstring: Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
AWS
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. See https://docs.databricks.com/aws/release-notes/serverless/#serverless-environment-versions.
Azure
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://learn.microsoft.com/azure/databricks/release-notes/serverless/#serverless-environment-versions.
GCP
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://docs.databricks.com/gcp/release-notes/serverless/#serverless-environment-versions.
Example: 5

java_dependenciesarray of string: List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
AWS
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar. See https://docs.databricks.com/aws/en/jobs/jar.
Azure
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://learn.microsoft.com/en-gb/azure/databricks/jobs/jar.
GCP
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://docs.databricks.com/gcp/en/jobs/jar.

budget_policy_idstringPublic Preview: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See effective_budget_policy_id for the budget policy used by this workload.
Example: 550e8400-e29b-41d4-a716-446655440000

performance_targetstring

The performance mode on a serverless job. This field determines the level of compute performance or cost-efficiency for the run. The performance target does not apply to tasks that run on Serverless GPU compute.

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Default: PERFORMANCE_OPTIMIZED

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

triggersarray of objectBeta

List of triggers attached to this job. A run starts when any active trigger evaluates to true. Cannot be set in the same request as the legacy schedule, trigger, or continuous fields. Gated behind the "Multiple Triggers" feature preview.

Show child attributesHide child attributes

pause_statusstringBeta: Whether this trigger is paused. Defaults to UNPAUSED when unset; the server always returns an explicit value on read.
Values: UNPAUSED, PAUSED

periodicobjectBeta

Trigger type: exactly one must be set; mutual exclusivity is enforced in the API handler Periodic trigger configuration.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

scheduleobjectBeta

Cron schedule trigger configuration.

Show child attributesHide child attributes

quartz_cron_expressionstringBeta: A Cron expression using Quartz syntax that describes the schedule for this trigger. See Cron Trigger for details.
Example: 20 30 * * * ?

timezone_idstringBeta: A Java timezone ID. The schedule is resolved with respect to this timezone. See Java TimeZone for details.
Example: Europe/London

continuousobjectBeta

Continuous trigger configuration.

Show child attributesHide child attributes

task_retry_modestringBeta: Whether the continuous job applies task-level retries. Defaults to NEVER.
Values: NEVER, ON_FAILURE

file_arrivalobjectBeta

File arrival trigger configuration.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

table_updateobjectBeta

Table update trigger configuration.

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

created_timeint64: The time at which this job was created in epoch milliseconds (milliseconds since 1/1/1970 UTC).
Example: 1601370337343

trigger_stateobject

State of the trigger associated with the job.

Show child attributesHide child attributes

tableobjectRequired

Show child attributesHide child attributes

last_seen_table_statesarray of object

Show child attributesHide child attributes

table_namestring: Full table name of the table to monitor, e.g. mycatalog.myschema.mytable

has_seen_updatesboolean: Whether or not the table has seen updates since either the creation of the trigger or the last successful evaluation of the trigger

using_scalable_monitoringboolean: Indicates whether the trigger is using scalable monitoring.

file_arrivalobjectRequired

Show child attributesHide child attributes

using_file_eventsboolean: Indicates whether the trigger leverages file events to detect file arrivals.

has_moreboolean: Indicates if the job has more array properties (tasks, job_clusters) that are not shown. They can be accessed via jobs/get endpoint. It is only relevant for API 2.2 jobs/list requests with expand_tasks=true.
Example: true

effective_budget_policy_idstringPublic Preview

The id of the budget policy used by this job for cost attribution purposes. This may be set through (in order of precedence):

Budget admins through the account or workspace console
Jobs UI in the job details page and Jobs API using budget_policy_id
Inferred default based on accessible budget policies of the run_as identity on job creation or modification.

trigger_detailsarray of objectBeta

Per-trigger runtime information for the multi-trigger surface. Same length and order as JobSettings.triggers; trigger_details[i] corresponds to triggers[i]. Sub-fields (state, history) are populated independently based on the GetJob.include_trigger_state / include_trigger_history flags.

Show child attributesHide child attributes

stateobjectBeta

Current runtime state. Populated when GetJob.include_trigger_state is set.

Show child attributesHide child attributes

periodicobjectRequiredBeta

Show child attributesHide child attributes

next_run_timeint64Beta

scheduleobjectRequiredBeta

continuousobjectRequiredBeta

Show child attributesHide child attributes

consecutive_failuresint32Beta

next_attempt_msint64Beta

is_backing_offbooleanBeta

file_arrivalobjectRequiredBeta

Show child attributesHide child attributes

using_file_eventsboolean: Indicates whether the trigger leverages file events to detect file arrivals.

table_updateobjectRequiredBeta

Show child attributesHide child attributes

last_seen_table_statesarray of object

using_scalable_monitoringboolean: Indicates whether the trigger is using scalable monitoring.

modelobjectRequiredBeta

pause_statusstringBeta: Whether this trigger is paused or not. Mirrors the configured pause_status.
Values: UNPAUSED, PAUSED

historyobjectBeta

Recent evaluation history. Populated when GetJob.include_trigger_history is set.

Show child attributesHide child attributes

last_triggeredobjectBeta

The last time the run was triggered due to a file arrival.

Show child attributesHide child attributes

timestampint64Beta: Timestamp at which the trigger was evaluated.
Example: 1642521600000

descriptionstringBeta: Human-readable description of the trigger evaluation result. Explains why the trigger evaluation triggered or did not trigger a run, or failed.
Example: Found 2 new files and 10 files in total

run_idint64Beta: The ID of the run that was triggered by the trigger evaluation. Only returned if a run was triggered.

last_not_triggeredobjectBeta

The last time the trigger was evaluated but did not trigger a run.

Show child attributesHide child attributes

timestampint64Beta: Timestamp at which the trigger was evaluated.
Example: 1642521600000

descriptionstringBeta: Human-readable description of the trigger evaluation result. Explains why the trigger evaluation triggered or did not trigger a run, or failed.
Example: Found 2 new files and 10 files in total

run_idint64Beta: The ID of the run that was triggered by the trigger evaluation. Only returned if a run was triggered.

last_failedobjectBeta

The last time the trigger failed to evaluate.

Show child attributesHide child attributes

timestampint64Beta: Timestamp at which the trigger was evaluated.
Example: 1642521600000

descriptionstringBeta: Human-readable description of the trigger evaluation result. Explains why the trigger evaluation triggered or did not trigger a run, or failed.
Example: Found 2 new files and 10 files in total

run_idint64Beta: The ID of the run that was triggered by the trigger evaluation. Only returned if a run was triggered.

View as Markdown

Response

{
  "next_page_token": "CAAos-uriYcxMN7_rt_v7B4=",
  "job_id": 11223344,
  "creator_user_name": "user.name@databricks.com",
  "run_as_user_name": "user.name@databricks.com",
  "settings": {
    "name": "A multitask job",
    "description": "This job contain multiple tasks that are required to produce the weekly shark sightings report.",
    "email_notifications": {},
    "webhook_notifications": {},
    "notification_settings": {},
    "timeout_seconds": 86400,
    "health": {},
    "schedule": {},
    "trigger": {},
    "continuous": {},
    "max_concurrent_runs": 10,
    "tasks": [
      {
        "depends_on": [],
        "description": "Extracts session data from events",
        "existing_cluster_id": "0923-164208-meows279",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/Sessionize.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.Sessionize",
          "parameters": [
            "--data",
            "dbfs:/path/to/data.json"
          ]
        },
        "task_key": "Sessionize",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [],
        "description": "Ingests order data",
        "job_cluster_key": "auto_scaling_cluster",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/OrderIngest.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.OrdersIngest",
          "parameters": [
            "--data",
            "dbfs:/path/to/order-data.json"
          ]
        },
        "task_key": "Orders_Ingest",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [
          {
            "task_key": "Orders_Ingest"
          },
          {
            "task_key": "Sessionize"
          }
        ],
        "description": "Matches orders with user sessions",
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        },
        "notebook_task": {
          "base_parameters": {
            "age": "35",
            "name": "John Doe"
          },
          "notebook_path": "/Users/user.name@databricks.com/Match"
        },
        "retry_on_timeout": false,
        "run_if": "ALL_SUCCESS",
        "task_key": "Match",
        "timeout_seconds": 86400
      }
    ],
    "job_clusters": [
      {
        "job_cluster_key": "auto_scaling_cluster",
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        }
      }
    ],
    "git_source": {},
    "tags": {
      "cost-center": "engineering",
      "team": "jobs"
    },
    "format": "MULTI_TASK",
    "queue": {},
    "parameters": [
      {}
    ],
    "run_as": {},
    "edit_mode": "string",
    "deployment": {},
    "environments": [
      {}
    ],
    "budget_policy_id": "550e8400-e29b-41d4-a716-446655440000",
    "performance_target": "string",
    "triggers": [
      {}
    ],
    "max_retries": 10,
    "min_retry_interval_millis": 2000,
    "retry_on_timeout": true,
    "disable_auto_optimization": true
  },
  "created_time": 1601370337343,
  "trigger_state": {
    "table": {},
    "file_arrival": {}
  },
  "has_more": true,
  "effective_budget_policy_id": "string",
  "trigger_details": [
    {
      "state": {},
      "history": {}
    }
  ]
}

List GA

GET /api/2.2/jobs/list

Retrieves a list of jobs.

API scopes: jobs

Parameters

limitint32query: The number of jobs to return. This value must be greater than 0 and less or equal to 100. The default value is 20.
Default: 20
Example: 25
Constraints: [ 1 .. 100 ]

expand_tasksbooleanquery: Whether to include task and cluster details in the response. Note that only the first 100 elements will be shown. Use jobs/get to paginate through all tasks and clusters.
Default: false
Example: false

namestringquery: A filter on the list based on the exact (case insensitive) job name.
Example: A%20multitask%20job

page_tokenstringquery: Use next_page_token or prev_page_token returned from the previous request to list the next or previous page of jobs respectively.
Example: CAEomPSriYcxMPWM_IiIxvEB

Response

jobsarray of object

The list of jobs. Only included in the response if there are jobs to list.

Show child attributesHide child attributes

job_idint64: The canonical identifier for this job.
Example: 11223344

creator_user_namestring: The creator user name. This field won’t be included in the response if the user has already been deleted.
Example: user.name@databricks.com

run_as_user_namestring

The email of an active workspace user or the application ID of a service principal that the job runs as. This value can be changed by setting the run_as field when creating or updating a job.

Example: user.name@databricks.com

settingsobject

Settings for this job and all of its runs. These settings can be updated using the resetJob method.

Show child attributesHide child attributes

namestring: An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.
Default: Untitled
Example: A multitask job
Constraints: <= 4096 characters

descriptionstring: An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.
Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.
Constraints: <= 27700 characters

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes

on_startarray of object: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

on_successarray of object: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

on_failurearray of object: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

on_duration_warning_threshold_exceededarray of object: An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

on_streaming_backlog_exceededarray of objectPublic Preview: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

timeout_secondsint32: An optional timeout applied to each run of this job. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

quartz_cron_expressionstring: A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
Example: 20 30 * * * ?

timezone_idstring: A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
Example: Europe/London

pause_statusstring: Indicate whether this schedule is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

triggerobject

Show child attributesHide child attributes

pause_statusstring: Whether this trigger is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

file_arrivalobjectRequired: File arrival trigger settings.

periodicobjectRequired: Periodic trigger settings.

table_updateobjectRequired

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes

pause_statusstring: Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

task_retry_modestring: Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.
Values: NEVER, ON_FAILURE

max_concurrent_runsint32: An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.
Default: 1
Example: 10

tasksarray of object

Show child attributesHide child attributes

task_keystring: A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

depends_onarray of object: An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the run_if condition is true. The key is task_key, and the value is the name assigned to the dependent task.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

ALL_SUCCESS: All dependencies have executed and succeeded
AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
NONE_FAILED: None of the dependencies have failed and at least one was executed
ALL_DONE: All dependencies have been completed
AT_LEAST_ONE_FAILED: At least one dependency failed
ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values: ALL_SUCCESS, ALL_DONE, NONE_FAILED, AT_LEAST_ONE_SUCCESS, ALL_FAILED, AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32: An optional timeout applied to each run of this job task. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

email_notificationsobject: An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

notification_settingsobject: Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

webhook_notificationsobject: A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

descriptionstring: An optional description for this task.
Example: This is the description for this task.
Constraints: <= 1000 characters

environment_keystringRequired: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

disabledboolean: An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
Default: false

computeobjectBeta: Task level compute configuration.

notebook_taskobjectRequired: The task runs a notebook when the notebook_task field is present.

spark_jar_taskobjectRequired: The task runs a JAR when the spark_jar_task field is present.

spark_python_taskobjectRequired: The task runs a Python file when the spark_python_task field is present.

spark_submit_taskobjectRequired: (Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

pipeline_taskobjectRequired: The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

python_wheel_taskobjectRequired: The task runs a Python wheel when the python_wheel_task field is present.

dbt_taskobjectRequired: The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

sql_taskobjectRequired: The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the sql_task field is present.

run_job_taskobjectRequired: The task triggers another job when the run_job_task field is present.

condition_taskobjectRequired: The task evaluates a condition that can be used to control the execution of other tasks when the condition_task field is present. The condition task does not require a cluster to execute and does not support retries or notifications.

for_each_taskobjectRequired: The task executes a nested task for every input provided when the for_each_task field is present.

clean_rooms_notebook_taskobjectRequired: The task runs a clean rooms notebook when the clean_rooms_notebook_task field is present.

alert_taskobjectRequiredPublic Preview: The task evaluates a <Databricks> alert and sends notifications to subscribers when the alert_task field is present.

power_bi_taskobjectRequiredPublic Preview: The task triggers a Power BI semantic model update when the power_bi_task field is present.

dashboard_taskobjectRequired: The task refreshes a dashboard and sends a snapshot to subscribers.

ai_runtime_taskobjectRequiredPublic Preview: The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.

existing_cluster_idstringRequired: If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
Example: 0923-164208-meows279

new_clusterobjectRequired: If new_cluster, a description of a new cluster that is created for each run.

job_cluster_keystringRequired: If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

librariesarray of object: An optional list of libraries to be installed on the cluster. The default value is an empty list.

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

job_clustersarray of object

A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings.

Constraints: <= 100 items

Show child attributesHide child attributes

job_cluster_keystring: A unique name for the job cluster. This field is required and must be unique within the job. JobTaskSettings may refer to this field to determine which cluster to launch for the task execution.
Example: auto_scaling_cluster
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

new_clusterobject: If new_cluster, a description of a cluster that is created for each task.

git_sourceobject

An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.

If git_source is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by setting source to WORKSPACE on the task.

Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used, git_source must be defined on the job.

Show child attributesHide child attributes

git_urlstring: URL of the repository to be cloned by this job.
Example: https://github.com/databricks/databricks-cli
Constraints: <= 300 characters

git_providerstring: Unique identifier of the service used to host the Git repository. The value is case insensitive.

git_branchstringRequired: Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
Example: main
Constraints: <= 255 characters

git_tagstringRequired: Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
Example: release-1.0.0
Constraints: <= 255 characters

git_commitstringRequired: Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
Example: e0056d01
Constraints: <= 64 characters

git_snapshotobject

sparse_checkoutobject

tagsobject: A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added to the job.

formatstring: Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is always set to "MULTI_TASK".
Values: SINGLE_TASK, MULTI_TASK
Example: MULTI_TASK

queueobject

The queue settings of the job.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

parametersarray of object

Job-level parameter definitions

Show child attributesHide child attributes

namestring: The name of the defined parameter. May only contain alphanumeric characters, _, -, and .
Example: table
Constraints: ^[\\w\\-.]+$

defaultstring: Default value of the parameter.
Example: users

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Non-admin users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

edit_modestring

Edit mode of the job.

UI_LOCKED: The job is in a locked UI state and cannot be modified.
EDITABLE: The job is in an editable state and can be modified.

Values: UI_LOCKED, EDITABLE

deploymentobject

Deployment information for jobs managed by external sources.

Show child attributesHide child attributes

kindstring

The kind of deployment that manages the job.

BUNDLE: The job is managed by Databricks Asset Bundle.
SYSTEM_MANAGED: The job is managed by <Databricks> and is read-only.

Values: BUNDLE, SYSTEM_MANAGED

metadata_file_pathstring: Path of the file that contains deployment metadata.

environmentsarray of object

Constraints: <= 10 items

Show child attributesHide child attributes

environment_keystring: The key of an environment. It has to be unique within a job.

specobject

budget_policy_idstringPublic Preview: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See effective_budget_policy_id for the budget policy used by this workload.
Example: 550e8400-e29b-41d4-a716-446655440000

performance_targetstring

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Default: PERFORMANCE_OPTIMIZED

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

triggersarray of objectBeta

Show child attributesHide child attributes

pause_statusstringBeta: Whether this trigger is paused. Defaults to UNPAUSED when unset; the server always returns an explicit value on read.
Values: UNPAUSED, PAUSED

periodicobjectBeta: Trigger type: exactly one must be set; mutual exclusivity is enforced in the API handler Periodic trigger configuration.

scheduleobjectBeta: Cron schedule trigger configuration.

continuousobjectBeta: Continuous trigger configuration.

file_arrivalobjectBeta: File arrival trigger configuration.

table_updateobjectBeta: Table update trigger configuration.

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

created_timeint64: The time at which this job was created in epoch milliseconds (milliseconds since 1/1/1970 UTC).
Example: 1601370337343

trigger_stateobject

State of the trigger associated with the job.

Show child attributesHide child attributes

tableobjectRequired

Show child attributesHide child attributes

last_seen_table_statesarray of object

using_scalable_monitoringboolean: Indicates whether the trigger is using scalable monitoring.

file_arrivalobjectRequired

Show child attributesHide child attributes

using_file_eventsboolean: Indicates whether the trigger leverages file events to detect file arrivals.

has_moreboolean: Indicates if the job has more array properties (tasks, job_clusters) that are not shown. They can be accessed via jobs/get endpoint. It is only relevant for API 2.2 jobs/list requests with expand_tasks=true.
Example: true

effective_budget_policy_idstringPublic Preview

The id of the budget policy used by this job for cost attribution purposes. This may be set through (in order of precedence):

Budget admins through the account or workspace console
Jobs UI in the job details page and Jobs API using budget_policy_id
Inferred default based on accessible budget policies of the run_as identity on job creation or modification.

trigger_detailsarray of objectBeta

Show child attributesHide child attributes

stateobjectBeta

Current runtime state. Populated when GetJob.include_trigger_state is set.

Show child attributesHide child attributes

periodicobjectRequiredBeta

scheduleobjectRequiredBeta

continuousobjectRequiredBeta

file_arrivalobjectRequiredBeta

table_updateobjectRequiredBeta

modelobjectRequiredBeta

pause_statusstringBeta: Whether this trigger is paused or not. Mirrors the configured pause_status.
Values: UNPAUSED, PAUSED

historyobjectBeta

Recent evaluation history. Populated when GetJob.include_trigger_history is set.

Show child attributesHide child attributes

last_triggeredobjectBeta: The last time the run was triggered due to a file arrival.

last_not_triggeredobjectBeta: The last time the trigger was evaluated but did not trigger a run.

last_failedobjectBeta: The last time the trigger failed to evaluate.

next_page_tokenstring: A token that can be used to list the next page of jobs (if applicable).
Example: CAEomPuciYcxMKbM9JvMlwU=

prev_page_tokenstring: A token that can be used to list the previous page of jobs (if applicable).
Example: CAAos-uriYcxMN7_rt_v7B4=

View as Markdown

Response

{
  "jobs": [
    {
      "job_id": 11223344,
      "creator_user_name": "user.name@databricks.com",
      "run_as_user_name": "user.name@databricks.com",
      "settings": {},
      "created_time": 1601370337343,
      "trigger_state": {},
      "has_more": true,
      "effective_budget_policy_id": "string",
      "trigger_details": [
        {}
      ]
    }
  ],
  "next_page_token": "CAEomPuciYcxMKbM9JvMlwU=",
  "prev_page_token": "CAAos-uriYcxMN7_rt_v7B4="
}

Create GA

POST /api/2.2/jobs/create

Create a new job.

API scopes: jobs

Request body

access_control_listarray of object

List of permissions to set on the job.

Show child attributesHide child attributes

user_namestringRequired

group_namestringRequired

service_principal_namestringRequired

permission_levelstring: Values: CAN_VIEW, CAN_MANAGE_RUN, IS_OWNER, CAN_MANAGE

namestring: An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.
Default: Untitled
Example: A multitask job
Constraints: <= 4096 characters

descriptionstring: An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.
Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.
Constraints: <= 27700 characters

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes

on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes

idstring

on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes

idstring

on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes

idstring

on_duration_warning_threshold_exceededarray of object

Show child attributesHide child attributes

idstring

on_streaming_backlog_exceededarray of objectPublic Preview

Show child attributesHide child attributes

idstring

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

timeout_secondsint32: An optional timeout applied to each run of this job. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

Show child attributesHide child attributes

metricstring: Values: RUN_DURATION_SECONDS, STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, STREAMING_BACKLOG_FILES
Example: RUN_DURATION_SECONDS

opstring: Values: GREATER_THAN
Example: GREATER_THAN

valueint64: Specifies the threshold value that the health metric should obey to satisfy the health rule.
Example: 10

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

quartz_cron_expressionstring: A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
Example: 20 30 * * * ?

timezone_idstring: A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
Example: Europe/London

pause_statusstring: Indicate whether this schedule is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

triggerobject

Show child attributesHide child attributes

pause_statusstring: Whether this trigger is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

file_arrivalobjectRequired

File arrival trigger settings.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

periodicobjectRequired

Periodic trigger settings.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

table_updateobjectRequired

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes

pause_statusstring: Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

task_retry_modestring: Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.
Values: NEVER, ON_FAILURE

max_concurrent_runsint32: An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.
Default: 1
Example: 10

tasksarray of object

Show child attributesHide child attributes

task_keystring: A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

depends_onarray of object

Show child attributesHide child attributes

task_keystring: The name of the task this task depends on.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

outcomestring: Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

ALL_SUCCESS: All dependencies have executed and succeeded
AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
NONE_FAILED: None of the dependencies have failed and at least one was executed
ALL_DONE: All dependencies have been completed
AT_LEAST_ONE_FAILED: At least one dependency failed
ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values: ALL_SUCCESS, ALL_DONE, NONE_FAILED, AT_LEAST_ONE_SUCCESS, ALL_FAILED, AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32: An optional timeout applied to each run of this job task. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

Show child attributesHide child attributes

metricstring: Values: RUN_DURATION_SECONDS, STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, STREAMING_BACKLOG_FILES
Example: RUN_DURATION_SECONDS

opstring: Values: GREATER_THAN
Example: GREATER_THAN

valueint64: Specifies the threshold value that the health metric should obey to satisfy the health rule.
Example: 10

email_notificationsobject

An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

Show child attributesHide child attributes

on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes

idstring

on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes

idstring

on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes

idstring

on_duration_warning_threshold_exceededarray of object

Show child attributesHide child attributes

idstring

on_streaming_backlog_exceededarray of objectPublic Preview

Show child attributesHide child attributes

idstring

descriptionstring: An optional description for this task.
Example: This is the description for this task.
Constraints: <= 1000 characters

environment_keystringRequired: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

disabledboolean: An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
Default: false

computeobjectBeta

Task level compute configuration.

Show child attributesHide child attributes

hardware_acceleratorstringBeta: Hardware accelerator configuration for Serverless GPU workloads.
Values: GPU_1X_A10, GPU_8X_H100

notebook_taskobjectRequired

The task runs a notebook when the notebook_task field is present.

Show child attributesHide child attributes

notebook_pathstring: The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
Example: /Users/user.name@databricks.com/notebook_to_run

base_parametersobject

If the notebook takes a parameter that is not specified in the job’s base_parameters or the run-now override parameters, the default value from the notebook is used.

Retrieve these parameters in a notebook using dbutils.widgets.get.

The JSON representation of this field cannot exceed 1MB.

sourcestring

WORKSPACE: Notebook is located in <Databricks> workspace.
GIT: Notebook is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

warehouse_idstring

Optional warehouse_id to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.

Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.

Example: ab12cd34efgh567i

spark_jar_taskobjectRequired

The task runs a JAR when the spark_jar_task field is present.

Show child attributesHide child attributes

jar_uristring

See the examples of classic and serverless compute usage at the top of the page.

main_class_namestring

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.

The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

Example: com.databricks.ComputeModels

parametersarray of string

Parameters passed to the main method.

Use Task parameter variables to set parameters containing information about job runs.

run_as_replboolean: Deprecated. A value of false is no longer supported.
Default: true

spark_python_taskobjectRequired

The task runs a Python file when the spark_python_task field is present.

Show child attributesHide child attributes

python_filestring: The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with /. For files stored in a remote repository, the path must be relative. This field is required.
Example: dbfs:/path/to/file.py

parametersarray of string

Command line parameters passed to the Python file.

Use Task parameter variables to set parameters containing information about job runs.

sourcestring

WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.
GIT: The Python file is located in a remote Git repository.

Values: WORKSPACE, GIT

Example: WORKSPACE

spark_submit_taskobjectRequired

(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

Show child attributesHide child attributes

parametersarray of string

Command-line parameters passed to spark submit.

Use Task parameter variables to set parameters containing information about job runs.

pipeline_taskobjectRequired

The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

Show child attributesHide child attributes

pipeline_idstring: The full name of the pipeline task to execute.
Example: a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5

parametersobjectBeta: Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

python_wheel_taskobjectRequired

The task runs a Python wheel when the python_wheel_task field is present.

Show child attributesHide child attributes

package_namestring: Name of the package to execute

entry_pointstring: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using $packageName.$entryPoint()

parametersarray of string: Command-line parameters passed to Python wheel task. Leave it empty if named_parameters is not null.

named_parametersobject: Command-line parameters passed to Python wheel task in the form of ["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty if parameters is not null.

dbt_taskobjectRequired

The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

Show child attributesHide child attributes

project_directorystring: Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.

commandsarray of string: A list of dbt commands to execute. All commands must start with dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.

schemastring: Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the default schema is used.

warehouse_idstring: ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the --profiles-dir command line argument.
Example: 30dade0507d960d1

profiles_directorystring: Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.

catalogstring: Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
Example: main

sourcestring

WORKSPACE: Project is located in <Databricks> workspace.
GIT: Project is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

sql_taskobjectRequired

The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the sql_task field is present.

Show child attributesHide child attributes

parametersobject: Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.

queryobjectRequired

If query, indicates that this job must execute a SQL query.

Show child attributesHide child attributes

query_idstringRequired: The canonical identifier of the SQL query.

dashboardobjectRequired

If dashboard, indicates that this job must refresh a SQL dashboard.

Show child attributesHide child attributes

dashboard_idstring: The canonical identifier of the SQL dashboard.

subscriptionsarray of object: If specified, dashboard snapshots are sent to subscriptions.

custom_subjectstring: Subject of the email sent to subscribers of this task.

pause_subscriptionsboolean: If true, the dashboard snapshot is not taken, and emails are not sent to subscribers.
Default: false

alertobjectRequired

If alert, indicates that this job must refresh a SQL alert.

Show child attributesHide child attributes

alert_idstring: The canonical identifier of the SQL alert.

subscriptionsarray of object: If specified, alert notifications are sent to subscribers.

pause_subscriptionsboolean: If true, the alert notifications are not sent to subscribers.
Default: false

fileobjectRequired

If file, indicates that this job runs a SQL file in a remote Git repository.

Show child attributesHide child attributes

pathstring: Path of the SQL file. Must be relative if the source is a remote Git repository and absolute for workspace paths.

sourcestring

Optional location type of the SQL file. When set to WORKSPACE, the SQL file will be retrieved from the local <Databricks> workspace. When set to GIT, the SQL file will be retrieved from a Git repository defined in git_source. If the value is empty, the task will use GIT if git_source is defined and WORKSPACE otherwise.

WORKSPACE: SQL file is located in <Databricks> workspace.
GIT: SQL file is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

warehouse_idstring: The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.

run_job_taskobjectRequired

The task triggers another job when the run_job_task field is present.

Show child attributesHide child attributes

job_idint64: ID of the job to trigger.

job_parametersobject: Job-level parameters used to trigger the job.

pipeline_paramsobject

Controls whether the pipeline should perform a full refresh

Show child attributesHide child attributes

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

condition_taskobjectRequired

Show child attributesHide child attributes

opstring

EQUAL_TO, NOT_EQUAL operators perform string comparison of their operands. This means that “12.0” == “12” will evaluate to false.
GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL operators perform numeric comparison of their operands. “12.0” >= “12” will evaluate to true, “10.0” >= “12” will evaluate to false.

Values: EQUAL_TO, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL, NOT_EQUAL

leftstring: The left operand of the condition task. Can be either a string value or a job state or parameter reference.

rightstring: The right operand of the condition task. Can be either a string value or a job state or parameter reference.

outcomestring: The condition expression evaluation result. Filled in if the task was successfully completed. Can be "true" or "false"

for_each_taskobjectRequired

The task executes a nested task for every input provided when the for_each_task field is present.

Show child attributesHide child attributes

inputsstring: Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
Constraints: <= 5000 characters

concurrencyint32: An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
Example: 20
Constraints: [ 1 .. 100 ]

taskobject: Configuration for the task that will be run for each element in the array

clean_rooms_notebook_taskobjectRequired

The task runs a clean rooms notebook when the clean_rooms_notebook_task field is present.

Show child attributesHide child attributes

clean_room_namestring: The clean room that the notebook belongs to.

notebook_namestring: Name of the notebook being run.

etagstring: Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the cleanroomassets/get API.

notebook_base_parametersobject: Base parameters to be used for the clean room notebook job.

alert_taskobjectRequiredPublic Preview

The task evaluates a <Databricks> alert and sends notifications to subscribers when the alert_task field is present.

Show child attributesHide child attributes

alert_idstringPublic Preview: The alert_id is the canonical identifier of the alert.

warehouse_idstringPublic Preview: The warehouse_id identifies the warehouse settings used by the alert task.

workspace_pathstringPublic Preview

The workspace_path is the path to the alert file in the workspace. The path:

must start with "/Workspace"
must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.

Example: /Workspace/Users/user@company.com/example.dbalert.json

subscribersarray of objectPublic Preview

The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.

Show child attributesHide child attributes

user_namestringRequiredPublic Preview: A valid workspace email address.
Example: user@databricks.com

destination_idstringRequiredPublic Preview: Example: ce8f875a-11ad-4117-bcc6-c5a542822aee

power_bi_taskobjectRequiredPublic Preview

The task triggers a Power BI semantic model update when the power_bi_task field is present.

Show child attributesHide child attributes

tablesarray of objectPublic Preview

The tables to be exported to Power BI

Show child attributesHide child attributes

namestringPublic Preview: The table name in <Databricks>

catalogstringPublic Preview: The catalog name in <Databricks>

schemastringPublic Preview: The schema name in <Databricks>

storage_modestringPublic Preview: The Power BI storage mode of the table
Values: DIRECT_QUERY, IMPORT, DUAL

warehouse_idstringPublic Preview: The SQL warehouse ID to use as the Power BI data source

power_bi_modelobjectPublic Preview

The semantic model to update

Show child attributesHide child attributes

workspace_namestringPublic Preview: The name of the Power BI workspace of the model

model_namestringPublic Preview: The name of the Power BI model

storage_modestringPublic Preview: The default storage mode of the Power BI model
Values: DIRECT_QUERY, IMPORT, DUAL

authentication_methodstringPublic Preview: How the published Power BI model authenticates to <Databricks>
Values: OAUTH, PAT

overwrite_existingbooleanPublic Preview: Whether to overwrite existing Power BI models

connection_resource_namestringPublic Preview: The resource name of the UC connection to authenticate from <Databricks> to Power BI

refresh_after_updatebooleanPublic Preview: Whether the model should be refreshed after the update

dashboard_taskobjectRequired

The task refreshes a dashboard and sends a snapshot to subscribers.

Show child attributesHide child attributes

subscriptionobject

Optional: subscription configuration for sending the dashboard snapshot.

Show child attributesHide child attributes

subscribersarray of object: The list of subscribers to send the snapshot of the dashboard to.

pausedboolean: When true, the subscription will not send emails.
Example: false

custom_subjectstring: Optional: Allows users to specify a custom subject line on the email sent to subscribers.
Example: Custom email subject
Constraints: <= 250 characters

warehouse_idstring: Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
Example: 47bb1c472649e711

dashboard_idstring: The identifier of the dashboard to refresh.
Example: 01ef0cb45e2a1da4a61950e9b8789ce9

ai_runtime_taskobjectRequiredPublic Preview

The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.

Show child attributesHide child attributes

experimentstringPublic Preview: MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set mlflow_experiment_directory.
Example: llama-fine-tune

deploymentsarray of objectPublic Preview

Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.

Show child attributesHide child attributes

command_pathstringPublic Preview

Workspace path of the script to run on each node in this deployment. Upload the script to this path and supply the path here. When the task runs, the file at this path is run on each node; if it fails, the task fails with its exit code.

Example script contents:

Plain Python:

python train.py --epochs 10

Multi-GPU via accelerate:

accelerate launch train.py --config config.yaml

Distributed via torchrun:

torchrun --nproc_per_node=8 train.py

Example: /Workspace/Users/alice@databricks.com/llama-fine-tune/command.sh

computeobjectPublic Preview: Compute resources allocated to each node in this deployment.

namestringPublic Preview: Optional human-readable name for this deployment (for example, driver, worker, param_server). Used for log and UI display. Distinct names are recommended so deployments can be told apart, but uniqueness is not enforced.
Example: worker

mlflow_runstringPublic Preview: Optional display name for the MLflow run created under experiment. If omitted, MLflow generates a default name.
Example: llama-fine-tune-lora-r16

mlflow_experiment_directorystringPublic Preview: Optional workspace directory under which the MLflow experiment named in experiment is created. Must start with /Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
Example: /Workspace/Users/alice@databricks.com/experiments

docker_image_urlstringBeta: Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format: {organization}/{repository}:{tag}
Example: nvidia/cuda:13.3.0-cudnn-devel-ubuntu24.04

existing_cluster_idstringRequired: If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
Example: 0923-164208-meows279

new_clusterobjectRequired

If new_cluster, a description of a new cluster that is created for each run.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, the zone "auto" will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

Nodes for this cluster will only be placed on AWS instances with this instance profile. If ommitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogenously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject: Defines values necessary to configure and run Azure Log Analytics agent

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Capacity reservations can only be specified when the workspace uses injected vnet (i.e. customer defined vnet not managed by databricks). Ensure the databricks-login-prod Enterprise Application is granted the following four permissions:

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

volumesobjectRequired: destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

fileobjectRequired: destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

gcsobjectRequired: destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

abfssobjectRequired: destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

workspaceobjectRequired: destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

volumesobjectRequired: destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired: Basic auth with username and password

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject: defined what type of clients can use the cluster. E.g. Notebooks, Jobs

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

job_cluster_keystringRequired: If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

librariesarray of object

An optional list of libraries to be installed on the cluster. The default value is an empty list.

Show child attributesHide child attributes

jarstringRequired: URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

eggstringRequired: Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.

pypiobjectRequired

Specification of a PyPi library to be installed. For example: { "package": "simplejson" }

Show child attributesHide child attributes

packagestring: The name of the pypi package to install. An optional exact version specification is also supported. Examples: "simplejson" and "simplejson==3.8.0".

repostring: The repository where the package can be found. If not specified, the default pip index is used.

mavenobjectRequired

Specification of a maven library to be installed. For example: { "coordinates": "org.jsoup:jsoup:1.7.2" }

Show child attributesHide child attributes

coordinatesstring: Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".

repostring: Maven repo to install the Maven package from. If omitted, both Maven Central Repository and Spark Packages are searched.

exclusionsarray of string

List of dependences to exclude. For example: ["slf4j:slf4j", "*:hadoop-client"].

Maven dependency exclusions: https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.

cranobjectRequired

Specification of a CRAN library to be installed as part of the library

Show child attributesHide child attributes

packagestring: The name of the CRAN package to install.

repostring: The repository where the package can be found. If not specified, the default CRAN repo is used.

whlstringRequired: URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

requirementsstringRequired: URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example: { "requirements": "/Workspace/path/to/requirements.txt" } or { "requirements" : "/Volumes/path/to/requirements.txt" }

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

job_clustersarray of object

A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings.

Constraints: <= 100 items

Show child attributesHide child attributes

job_cluster_keystring: A unique name for the job cluster. This field is required and must be unique within the job. JobTaskSettings may refer to this field to determine which cluster to launch for the task execution.
Example: auto_scaling_cluster
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

new_clusterobject

If new_cluster, a description of a cluster that is created for each task.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 0

availabilitystring: Default: SPOT_WITH_FALLBACK
Values: SPOT, ON_DEMAND, SPOT_WITH_FALLBACK

zone_idstring

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

This feature may only be available to certain customer plans.

spot_bid_price_percentint32: The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.
Default: 100

ebs_volume_typestring: The type of EBS volumes that will be launched with this cluster.
Values: GENERAL_PURPOSE_SSD, THROUGHPUT_OPTIMIZED_HDD

ebs_volume_countint32

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

Default: 0

ebs_volume_sizeint32: The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32: If using gp3 volumes, what IOPS to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

ebs_volume_throughputint32: If using gp3 volumes, what throughput to use for the disk. If this is not set, the maximum performance of a gp2 volume with the same volume size will be used.

azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

log_analytics_infoobject: Defines values necessary to configure and run Azure Log Analytics agent

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

availabilitystring: Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.
Default: ON_DEMAND_AZURE
Values: SPOT_AZURE, ON_DEMAND_AZURE, SPOT_WITH_FALLBACK_AZURE

spot_bid_max_pricedouble: The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.
Default: -1

capacity_reservation_groupstring

The Azure capacity reservation group resource ID to use for launching VMs. When specified, VMs will be launched using the provided capacity reservation.

Microsoft.Compute/capacityReservationGroups/read
Microsoft.Compute/capacityReservationGroups/deploy/action
Microsoft.Compute/capacityReservationGroups/capacityReservations/read
Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Format: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/capacityReservationGroups/{capacityReservationGroupName}

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes

use_preemptible_executorsboolean: This field determines whether the spark executors will be scheduled to run on preemptible VMs (when set to true) versus standard compute engine VMs (when set to false; default). Note: Soon to be deprecated, use the 'availability' field instead.
Default: false

google_service_accountstring: If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32: Boot disk size in GB

availabilitystring: This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.
Default: ON_DEMAND_GCP
Values: PREEMPTIBLE_GCP, ON_DEMAND_GCP, PREEMPTIBLE_WITH_FALLBACK_GCP

zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

"HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
"AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.

Default: HA

local_ssd_countint32: If provided, each node (workers and driver) in the cluster will have this number of local SSDs attached. Each local SSD is 375GB in size. Refer to GCP documentation for the supported number of local SSDs for each instance type.

first_on_demandint32: The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.
Default: 1

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject

Flexible node type configuration for worker nodes.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

driver_node_type_flexibilityobject

Flexible node type configuration for the driver node.

Show child attributesHide child attributes

alternate_node_type_idsarray of string: A list of node type IDs to use as fallbacks when the primary node type is unavailable.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { "s3": { "destination" : "s3://cluster_log_bucket/prefix", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

volumesobjectRequired: destination needs to be provided, e.g. { "volumes": { "destination": "/Volumes/catalog/schema/volume/cluster_log" } }

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object

Show child attributesHide child attributes

dbfsobjectRequired: destination needs to be provided. e.g. { "dbfs": { "destination" : "dbfs:/home/cluster_log" } }

s3objectRequired: destination and either the region or endpoint need to be provided. e.g. { \"s3\": { \"destination\": \"s3://cluster_log_bucket/prefix\", \"region\": \"us-west-2\" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

fileobjectRequired: destination needs to be provided, e.g. { "file": { "destination": "file:/my/local/file.sh" } }

gcsobjectRequired: destination needs to be provided, e.g. { "gcs": { "destination": "gs://my-bucket/file.sh" } }

abfssobjectRequired: destination needs to be provided, e.g. abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<directory-name>

workspaceobjectRequired: destination needs to be provided, e.g. { "workspace": { "destination": "/cluster-init-scripts/setup-datadog.sh" } }

volumesobjectRequired: destination needs to be provided. e.g. { \"volumes\" : { \"destination\" : \"/Volumes/my-init.sh\" } }

docker_imageobject

Custom docker image BYOC

Show child attributesHide child attributes

urlstring: URL of the docker image.

basic_authobjectRequired: Basic auth with username and password

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

Show child attributesHide child attributes

clientsobject: defined what type of clients can use the cluster. E.g. Notebooks, Jobs

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes

min_workersint32: The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32: The maximum number of workers to which the cluster can scale up when overloaded. Note that max_workers must be strictly greater than min_workers.

git_sourceobject

An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.

If git_source is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by setting source to WORKSPACE on the task.

Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used, git_source must be defined on the job.

Show child attributesHide child attributes

git_urlstring: URL of the repository to be cloned by this job.
Example: https://github.com/databricks/databricks-cli
Constraints: <= 300 characters

git_providerstring: Unique identifier of the service used to host the Git repository. The value is case insensitive.

git_branchstringRequired: Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
Example: main
Constraints: <= 255 characters

git_tagstringRequired: Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
Example: release-1.0.0
Constraints: <= 255 characters

git_commitstringRequired: Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
Example: e0056d01
Constraints: <= 64 characters

git_snapshotobject

Show child attributesHide child attributes

used_commitstring: Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
Example: 4506fdf41e9fa98090570a34df7a5bce163ff15f

sparse_checkoutobject

Show child attributesHide child attributes

patternsarray of string: List of patterns to include for sparse checkout.

tagsobject: A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added to the job.

formatstring: Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is always set to "MULTI_TASK".
Values: SINGLE_TASK, MULTI_TASK
Example: MULTI_TASK

queueobject

The queue settings of the job.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

parametersarray of object

Job-level parameter definitions

Show child attributesHide child attributes

namestring: The name of the defined parameter. May only contain alphanumeric characters, _, -, and .
Example: table
Constraints: ^[\\w\\-.]+$

defaultstring: Default value of the parameter.
Example: users

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Non-admin users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

edit_modestring

Edit mode of the job.

UI_LOCKED: The job is in a locked UI state and cannot be modified.
EDITABLE: The job is in an editable state and can be modified.

Values: UI_LOCKED, EDITABLE

deploymentobject

Deployment information for jobs managed by external sources.

Show child attributesHide child attributes

kindstring

The kind of deployment that manages the job.

BUNDLE: The job is managed by Databricks Asset Bundle.
SYSTEM_MANAGED: The job is managed by <Databricks> and is read-only.

Values: BUNDLE, SYSTEM_MANAGED

metadata_file_pathstring: Path of the file that contains deployment metadata.

environmentsarray of object

Constraints: <= 10 items

Show child attributesHide child attributes

environment_keystring: The key of an environment. It has to be unique within a job.

specobject

Show child attributesHide child attributes

clientstring: Use environment_version instead.
Example: 1

dependenciesarray of string: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a valid pip requirements file line per https://pip.pypa.io/en/stable/reference/requirements-file-format/. Allowed dependencies include a requirement specifier, an archive URL, a local project path (such as WSFS or UC Volumes in <Databricks>), or a VCS project URL.

base_environmentstring: The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
AWS
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see https://docs.databricks.com/aws/en/compute/serverless/dependencies#base-environments-for-job-tasks.
Azure
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://learn.microsoft.com/azure/databricks/compute/serverless/dependencies#base-environments-for-job-tasks.
GCP
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://docs.databricks.com/gcp/en/compute/serverless/dependencies#base-environments-for-job-tasks.

environment_versionstring: Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
AWS
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. See https://docs.databricks.com/aws/release-notes/serverless/#serverless-environment-versions.
Azure
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://learn.microsoft.com/azure/databricks/release-notes/serverless/#serverless-environment-versions.
GCP
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://docs.databricks.com/gcp/release-notes/serverless/#serverless-environment-versions.
Example: 5

java_dependenciesarray of string: List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
AWS
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar. See https://docs.databricks.com/aws/en/jobs/jar.
Azure
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://learn.microsoft.com/en-gb/azure/databricks/jobs/jar.
GCP
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://docs.databricks.com/gcp/en/jobs/jar.

budget_policy_idstringPublic Preview: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See effective_budget_policy_id for the budget policy used by this workload.
Example: 550e8400-e29b-41d4-a716-446655440000

performance_targetstring

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Default: PERFORMANCE_OPTIMIZED

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

triggersarray of objectBeta

Show child attributesHide child attributes

pause_statusstringBeta: Whether this trigger is paused. Defaults to UNPAUSED when unset; the server always returns an explicit value on read.
Values: UNPAUSED, PAUSED

periodicobjectBeta

Trigger type: exactly one must be set; mutual exclusivity is enforced in the API handler Periodic trigger configuration.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

scheduleobjectBeta

Cron schedule trigger configuration.

Show child attributesHide child attributes

quartz_cron_expressionstringBeta: A Cron expression using Quartz syntax that describes the schedule for this trigger. See Cron Trigger for details.
Example: 20 30 * * * ?

timezone_idstringBeta: A Java timezone ID. The schedule is resolved with respect to this timezone. See Java TimeZone for details.
Example: Europe/London

continuousobjectBeta

Continuous trigger configuration.

Show child attributesHide child attributes

task_retry_modestringBeta: Whether the continuous job applies task-level retries. Defaults to NEVER.
Values: NEVER, ON_FAILURE

file_arrivalobjectBeta

File arrival trigger configuration.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

table_updateobjectBeta

Table update trigger configuration.

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

Response

job_idint64: The canonical identifier for the newly created job.
Example: 11223344

View as Markdown

Request

Create a multi-task job on a shared job cluster

{
  "name": "Nightly ETL",
  "tags": {
    "cost-center": "engineering",
    "pipeline": "etl"
  },
  "job_clusters": [
    {
      "job_cluster_key": "shared",
      "new_cluster": {
        "spark_version": "13.3.x-scala2.12",
        "node_type_id": "i3.xlarge",
        "num_workers": 2
      }
    }
  ],
  "tasks": [
    {
      "task_key": "ingest",
      "description": "Ingest raw records into the bronze table",
      "job_cluster_key": "shared",
      "notebook_task": {
        "notebook_path": "/Repos/etl/ingest",
        "base_parameters": {
          "run_date": "2024-01-01"
        }
      }
    },
    {
      "task_key": "transform",
      "description": "Transform bronze into the silver table",
      "job_cluster_key": "shared",
      "depends_on": [
        {
          "task_key": "ingest"
        }
      ],
      "notebook_task": {
        "notebook_path": "/Repos/etl/transform"
      }
    }
  ],
  "queue": {
    "enabled": true
  },
  "max_concurrent_runs": 1
}

Response

{
  "job_id": 11223344
}

Update GA

POST /api/2.2/jobs/reset

Overwrite all settings for the given job. Use the Update endpoint to update job settings partially.

API scopes: jobs

Request body

job_idint64: The canonical identifier of the job to reset. This field is required.
Example: 11223344

new_settingsobject

The new settings of the job. These settings completely replace the old settings.

Changes to the field JobBaseSettings.timeout_seconds are applied to active runs. Changes to other fields are applied to future runs only.

Show child attributesHide child attributes

namestring: An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.
Default: Untitled
Example: A multitask job
Constraints: <= 4096 characters

descriptionstring: An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.
Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.
Constraints: <= 27700 characters

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes

on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes

idstring

on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes

idstring

on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes

idstring

on_duration_warning_threshold_exceededarray of object

Show child attributesHide child attributes

idstring

on_streaming_backlog_exceededarray of objectPublic Preview

Show child attributesHide child attributes

idstring

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

timeout_secondsint32: An optional timeout applied to each run of this job. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

Show child attributesHide child attributes

metricstring: Values: RUN_DURATION_SECONDS, STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, STREAMING_BACKLOG_FILES
Example: RUN_DURATION_SECONDS

opstring: Values: GREATER_THAN
Example: GREATER_THAN

valueint64: Specifies the threshold value that the health metric should obey to satisfy the health rule.
Example: 10

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

quartz_cron_expressionstring: A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
Example: 20 30 * * * ?

timezone_idstring: A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
Example: Europe/London

pause_statusstring: Indicate whether this schedule is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

triggerobject

Show child attributesHide child attributes

pause_statusstring: Whether this trigger is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

file_arrivalobjectRequired

File arrival trigger settings.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

periodicobjectRequired

Periodic trigger settings.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

table_updateobjectRequired

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes

pause_statusstring: Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

task_retry_modestring: Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.
Values: NEVER, ON_FAILURE

max_concurrent_runsint32: An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.
Default: 1
Example: 10

tasksarray of object

Show child attributesHide child attributes

task_keystring: A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

depends_onarray of object

Show child attributesHide child attributes

task_keystring: The name of the task this task depends on.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

outcomestring: Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

ALL_SUCCESS: All dependencies have executed and succeeded
AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
NONE_FAILED: None of the dependencies have failed and at least one was executed
ALL_DONE: All dependencies have been completed
AT_LEAST_ONE_FAILED: At least one dependency failed
ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values: ALL_SUCCESS, ALL_DONE, NONE_FAILED, AT_LEAST_ONE_SUCCESS, ALL_FAILED, AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32: An optional timeout applied to each run of this job task. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

email_notificationsobject

An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

Show child attributesHide child attributes

on_startarray of object: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

on_successarray of object: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

on_failurearray of object: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

on_duration_warning_threshold_exceededarray of object: An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

on_streaming_backlog_exceededarray of objectPublic Preview: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

descriptionstring: An optional description for this task.
Example: This is the description for this task.
Constraints: <= 1000 characters

environment_keystringRequired: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

disabledboolean: An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
Default: false

computeobjectBeta

Task level compute configuration.

Show child attributesHide child attributes

hardware_acceleratorstringBeta: Hardware accelerator configuration for Serverless GPU workloads.
Values: GPU_1X_A10, GPU_8X_H100

notebook_taskobjectRequired

The task runs a notebook when the notebook_task field is present.

Show child attributesHide child attributes

notebook_pathstring: The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
Example: /Users/user.name@databricks.com/notebook_to_run

base_parametersobject

If the notebook takes a parameter that is not specified in the job’s base_parameters or the run-now override parameters, the default value from the notebook is used.

Retrieve these parameters in a notebook using dbutils.widgets.get.

The JSON representation of this field cannot exceed 1MB.

sourcestring

WORKSPACE: Notebook is located in <Databricks> workspace.
GIT: Notebook is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

warehouse_idstring

Optional warehouse_id to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.

Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.

Example: ab12cd34efgh567i

spark_jar_taskobjectRequired

The task runs a JAR when the spark_jar_task field is present.

Show child attributesHide child attributes

jar_uristring

See the examples of classic and serverless compute usage at the top of the page.

main_class_namestring

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.

The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

Example: com.databricks.ComputeModels

parametersarray of string

Parameters passed to the main method.

Use Task parameter variables to set parameters containing information about job runs.

run_as_replboolean: Deprecated. A value of false is no longer supported.
Default: true

spark_python_taskobjectRequired

The task runs a Python file when the spark_python_task field is present.

Show child attributesHide child attributes

python_filestring: The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with /. For files stored in a remote repository, the path must be relative. This field is required.
Example: dbfs:/path/to/file.py

parametersarray of string

Command line parameters passed to the Python file.

Use Task parameter variables to set parameters containing information about job runs.

sourcestring

WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.
GIT: The Python file is located in a remote Git repository.

Values: WORKSPACE, GIT

Example: WORKSPACE

spark_submit_taskobjectRequired

(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

Show child attributesHide child attributes

parametersarray of string

Command-line parameters passed to spark submit.

Use Task parameter variables to set parameters containing information about job runs.

pipeline_taskobjectRequired

The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

Show child attributesHide child attributes

pipeline_idstring: The full name of the pipeline task to execute.
Example: a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5

parametersobjectBeta: Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

python_wheel_taskobjectRequired

The task runs a Python wheel when the python_wheel_task field is present.

Show child attributesHide child attributes

package_namestring: Name of the package to execute

entry_pointstring: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using $packageName.$entryPoint()

parametersarray of string: Command-line parameters passed to Python wheel task. Leave it empty if named_parameters is not null.

named_parametersobject: Command-line parameters passed to Python wheel task in the form of ["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty if parameters is not null.

dbt_taskobjectRequired

The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

Show child attributesHide child attributes

project_directorystring: Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.

commandsarray of string: A list of dbt commands to execute. All commands must start with dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.

schemastring: Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the default schema is used.

warehouse_idstring: ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the --profiles-dir command line argument.
Example: 30dade0507d960d1

profiles_directorystring: Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.

catalogstring: Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
Example: main

sourcestring

WORKSPACE: Project is located in <Databricks> workspace.
GIT: Project is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

sql_taskobjectRequired

The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the sql_task field is present.

Show child attributesHide child attributes

parametersobject: Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.

queryobjectRequired: If query, indicates that this job must execute a SQL query.

dashboardobjectRequired: If dashboard, indicates that this job must refresh a SQL dashboard.

alertobjectRequired: If alert, indicates that this job must refresh a SQL alert.

fileobjectRequired: If file, indicates that this job runs a SQL file in a remote Git repository.

warehouse_idstring: The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.

run_job_taskobjectRequired

The task triggers another job when the run_job_task field is present.

Show child attributesHide child attributes

job_idint64: ID of the job to trigger.

job_parametersobject: Job-level parameters used to trigger the job.

pipeline_paramsobject: Controls whether the pipeline should perform a full refresh

condition_taskobjectRequired

Show child attributesHide child attributes

opstring

EQUAL_TO, NOT_EQUAL operators perform string comparison of their operands. This means that “12.0” == “12” will evaluate to false.
GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL operators perform numeric comparison of their operands. “12.0” >= “12” will evaluate to true, “10.0” >= “12” will evaluate to false.

Values: EQUAL_TO, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL, NOT_EQUAL

leftstring: The left operand of the condition task. Can be either a string value or a job state or parameter reference.

rightstring: The right operand of the condition task. Can be either a string value or a job state or parameter reference.

outcomestring: The condition expression evaluation result. Filled in if the task was successfully completed. Can be "true" or "false"

for_each_taskobjectRequired

The task executes a nested task for every input provided when the for_each_task field is present.

Show child attributesHide child attributes

inputsstring: Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
Constraints: <= 5000 characters

concurrencyint32: An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
Example: 20
Constraints: [ 1 .. 100 ]

taskobject: Configuration for the task that will be run for each element in the array

clean_rooms_notebook_taskobjectRequired

The task runs a clean rooms notebook when the clean_rooms_notebook_task field is present.

Show child attributesHide child attributes

clean_room_namestring: The clean room that the notebook belongs to.

notebook_namestring: Name of the notebook being run.

etagstring: Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the cleanroomassets/get API.

notebook_base_parametersobject: Base parameters to be used for the clean room notebook job.

alert_taskobjectRequiredPublic Preview

The task evaluates a <Databricks> alert and sends notifications to subscribers when the alert_task field is present.

Show child attributesHide child attributes

alert_idstringPublic Preview: The alert_id is the canonical identifier of the alert.

warehouse_idstringPublic Preview: The warehouse_id identifies the warehouse settings used by the alert task.

workspace_pathstringPublic Preview

The workspace_path is the path to the alert file in the workspace. The path:

must start with "/Workspace"
must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.

Example: /Workspace/Users/user@company.com/example.dbalert.json

subscribersarray of objectPublic Preview: The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.

power_bi_taskobjectRequiredPublic Preview

The task triggers a Power BI semantic model update when the power_bi_task field is present.

Show child attributesHide child attributes

tablesarray of objectPublic Preview: The tables to be exported to Power BI

warehouse_idstringPublic Preview: The SQL warehouse ID to use as the Power BI data source

power_bi_modelobjectPublic Preview: The semantic model to update

connection_resource_namestringPublic Preview: The resource name of the UC connection to authenticate from <Databricks> to Power BI

refresh_after_updatebooleanPublic Preview: Whether the model should be refreshed after the update

dashboard_taskobjectRequired

The task refreshes a dashboard and sends a snapshot to subscribers.

Show child attributesHide child attributes

subscriptionobject: Optional: subscription configuration for sending the dashboard snapshot.

warehouse_idstring: Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
Example: 47bb1c472649e711

dashboard_idstring: The identifier of the dashboard to refresh.
Example: 01ef0cb45e2a1da4a61950e9b8789ce9

ai_runtime_taskobjectRequiredPublic Preview

The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.

Show child attributesHide child attributes

experimentstringPublic Preview: MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set mlflow_experiment_directory.
Example: llama-fine-tune

deploymentsarray of objectPublic Preview: Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.

mlflow_runstringPublic Preview: Optional display name for the MLflow run created under experiment. If omitted, MLflow generates a default name.
Example: llama-fine-tune-lora-r16

mlflow_experiment_directorystringPublic Preview: Optional workspace directory under which the MLflow experiment named in experiment is created. Must start with /Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
Example: /Workspace/Users/alice@databricks.com/experiments

docker_image_urlstringBeta: Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format: {organization}/{repository}:{tag}
Example: nvidia/cuda:13.3.0-cudnn-devel-ubuntu24.04

existing_cluster_idstringRequired: If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
Example: 0923-164208-meows279

new_clusterobjectRequired

If new_cluster, a description of a new cluster that is created for each run.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

job_cluster_keystringRequired: If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

librariesarray of object

An optional list of libraries to be installed on the cluster. The default value is an empty list.

Show child attributesHide child attributes

jarstringRequired: URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

eggstringRequired: Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.

pypiobjectRequired: Specification of a PyPi library to be installed. For example: { "package": "simplejson" }

mavenobjectRequired: Specification of a maven library to be installed. For example: { "coordinates": "org.jsoup:jsoup:1.7.2" }

cranobjectRequired: Specification of a CRAN library to be installed as part of the library

whlstringRequired: URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

requirementsstringRequired: URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example: { "requirements": "/Workspace/path/to/requirements.txt" } or { "requirements" : "/Volumes/path/to/requirements.txt" }

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

job_clustersarray of object

A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings.

Constraints: <= 100 items

Show child attributesHide child attributes

job_cluster_keystring: A unique name for the job cluster. This field is required and must be unique within the job. JobTaskSettings may refer to this field to determine which cluster to launch for the task execution.
Example: auto_scaling_cluster
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

new_clusterobject

If new_cluster, a description of a cluster that is created for each task.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

git_sourceobject

An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.

If git_source is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by setting source to WORKSPACE on the task.

Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used, git_source must be defined on the job.

Show child attributesHide child attributes

git_urlstring: URL of the repository to be cloned by this job.
Example: https://github.com/databricks/databricks-cli
Constraints: <= 300 characters

git_providerstring: Unique identifier of the service used to host the Git repository. The value is case insensitive.

git_branchstringRequired: Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
Example: main
Constraints: <= 255 characters

git_tagstringRequired: Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
Example: release-1.0.0
Constraints: <= 255 characters

git_commitstringRequired: Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
Example: e0056d01
Constraints: <= 64 characters

git_snapshotobject

Show child attributesHide child attributes

used_commitstring: Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
Example: 4506fdf41e9fa98090570a34df7a5bce163ff15f

sparse_checkoutobject

Show child attributesHide child attributes

patternsarray of string: List of patterns to include for sparse checkout.

tagsobject: A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added to the job.

formatstring: Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is always set to "MULTI_TASK".
Values: SINGLE_TASK, MULTI_TASK
Example: MULTI_TASK

queueobject

The queue settings of the job.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

parametersarray of object

Job-level parameter definitions

Show child attributesHide child attributes

namestring: The name of the defined parameter. May only contain alphanumeric characters, _, -, and .
Example: table
Constraints: ^[\\w\\-.]+$

defaultstring: Default value of the parameter.
Example: users

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Non-admin users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

edit_modestring

Edit mode of the job.

UI_LOCKED: The job is in a locked UI state and cannot be modified.
EDITABLE: The job is in an editable state and can be modified.

Values: UI_LOCKED, EDITABLE

deploymentobject

Deployment information for jobs managed by external sources.

Show child attributesHide child attributes

kindstring

The kind of deployment that manages the job.

BUNDLE: The job is managed by Databricks Asset Bundle.
SYSTEM_MANAGED: The job is managed by <Databricks> and is read-only.

Values: BUNDLE, SYSTEM_MANAGED

metadata_file_pathstring: Path of the file that contains deployment metadata.

environmentsarray of object

Constraints: <= 10 items

Show child attributesHide child attributes

environment_keystring: The key of an environment. It has to be unique within a job.

specobject

Show child attributesHide child attributes

clientstring: Use environment_version instead.
Example: 1

dependenciesarray of string: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a valid pip requirements file line per https://pip.pypa.io/en/stable/reference/requirements-file-format/. Allowed dependencies include a requirement specifier, an archive URL, a local project path (such as WSFS or UC Volumes in <Databricks>), or a VCS project URL.

base_environmentstring: The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
AWS
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see https://docs.databricks.com/aws/en/compute/serverless/dependencies#base-environments-for-job-tasks.
Azure
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://learn.microsoft.com/azure/databricks/compute/serverless/dependencies#base-environments-for-job-tasks.
GCP
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://docs.databricks.com/gcp/en/compute/serverless/dependencies#base-environments-for-job-tasks.

environment_versionstring: Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
AWS
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. See https://docs.databricks.com/aws/release-notes/serverless/#serverless-environment-versions.
Azure
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://learn.microsoft.com/azure/databricks/release-notes/serverless/#serverless-environment-versions.
GCP
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://docs.databricks.com/gcp/release-notes/serverless/#serverless-environment-versions.
Example: 5

java_dependenciesarray of string: List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
AWS
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar. See https://docs.databricks.com/aws/en/jobs/jar.
Azure
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://learn.microsoft.com/en-gb/azure/databricks/jobs/jar.
GCP
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://docs.databricks.com/gcp/en/jobs/jar.

budget_policy_idstringPublic Preview: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See effective_budget_policy_id for the budget policy used by this workload.
Example: 550e8400-e29b-41d4-a716-446655440000

performance_targetstring

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Default: PERFORMANCE_OPTIMIZED

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

triggersarray of objectBeta

Show child attributesHide child attributes

pause_statusstringBeta: Whether this trigger is paused. Defaults to UNPAUSED when unset; the server always returns an explicit value on read.
Values: UNPAUSED, PAUSED

periodicobjectBeta

Trigger type: exactly one must be set; mutual exclusivity is enforced in the API handler Periodic trigger configuration.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

scheduleobjectBeta

Cron schedule trigger configuration.

Show child attributesHide child attributes

quartz_cron_expressionstringBeta: A Cron expression using Quartz syntax that describes the schedule for this trigger. See Cron Trigger for details.
Example: 20 30 * * * ?

timezone_idstringBeta: A Java timezone ID. The schedule is resolved with respect to this timezone. See Java TimeZone for details.
Example: Europe/London

continuousobjectBeta

Continuous trigger configuration.

Show child attributesHide child attributes

task_retry_modestringBeta: Whether the continuous job applies task-level retries. Defaults to NEVER.
Values: NEVER, ON_FAILURE

file_arrivalobjectBeta

File arrival trigger configuration.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

table_updateobjectBeta

Table update trigger configuration.

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

View as Markdown

Request

{
  "job_id": 11223344,
  "new_settings": {
    "name": "A multitask job",
    "description": "This job contain multiple tasks that are required to produce the weekly shark sightings report.",
    "email_notifications": {},
    "webhook_notifications": {},
    "notification_settings": {},
    "timeout_seconds": 86400,
    "health": {},
    "schedule": {},
    "trigger": {},
    "continuous": {},
    "max_concurrent_runs": 10,
    "tasks": [
      {
        "depends_on": [],
        "description": "Extracts session data from events",
        "existing_cluster_id": "0923-164208-meows279",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/Sessionize.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.Sessionize",
          "parameters": [
            "--data",
            "dbfs:/path/to/data.json"
          ]
        },
        "task_key": "Sessionize",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [],
        "description": "Ingests order data",
        "job_cluster_key": "auto_scaling_cluster",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/OrderIngest.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.OrdersIngest",
          "parameters": [
            "--data",
            "dbfs:/path/to/order-data.json"
          ]
        },
        "task_key": "Orders_Ingest",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [
          {
            "task_key": "Orders_Ingest"
          },
          {
            "task_key": "Sessionize"
          }
        ],
        "description": "Matches orders with user sessions",
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        },
        "notebook_task": {
          "base_parameters": {
            "age": "35",
            "name": "John Doe"
          },
          "notebook_path": "/Users/user.name@databricks.com/Match"
        },
        "retry_on_timeout": false,
        "run_if": "ALL_SUCCESS",
        "task_key": "Match",
        "timeout_seconds": 86400
      }
    ],
    "job_clusters": [
      {
        "job_cluster_key": "auto_scaling_cluster",
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        }
      }
    ],
    "git_source": {},
    "tags": {
      "cost-center": "engineering",
      "team": "jobs"
    },
    "format": "MULTI_TASK",
    "queue": {},
    "parameters": [
      {}
    ],
    "run_as": {},
    "edit_mode": "string",
    "deployment": {},
    "environments": [
      {}
    ],
    "budget_policy_id": "550e8400-e29b-41d4-a716-446655440000",
    "performance_target": "string",
    "triggers": [
      {}
    ],
    "max_retries": 10,
    "min_retry_interval_millis": 2000,
    "retry_on_timeout": true,
    "disable_auto_optimization": true
  }
}

Response

{}

Delete GA

POST /api/2.2/jobs/delete

Deletes a job.

API scopes: jobs

Request body

job_idint64: The canonical identifier of the job to delete. This field is required.
Example: 11223344

Update Job GA

POST /api/2.2/jobs/update

Add, update, or remove specific settings of an existing job. Use the Reset endpoint to overwrite all job settings.

API scopes: jobs

Request body

job_idint64: The canonical identifier of the job to update. This field is required.
Example: 11223344

new_settingsobject

The new settings for the job.

Top-level fields specified in new_settings are completely replaced, except for arrays which are merged. That is, new and existing entries are completely replaced based on the respective key fields, i.e. task_key or job_cluster_key, while previous entries are kept.

Partially updating nested fields is not supported.

Changes to the field JobSettings.timeout_seconds are applied to active runs. Changes to other fields are applied to future runs only.

Show child attributesHide child attributes

namestring: An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.
Default: Untitled
Example: A multitask job
Constraints: <= 4096 characters

descriptionstring: An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.
Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.
Constraints: <= 27700 characters

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes

on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes

idstring

on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes

idstring

on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes

idstring

on_duration_warning_threshold_exceededarray of object

Show child attributesHide child attributes

idstring

on_streaming_backlog_exceededarray of objectPublic Preview

Show child attributesHide child attributes

idstring

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

timeout_secondsint32: An optional timeout applied to each run of this job. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

Show child attributesHide child attributes

metricstring: Values: RUN_DURATION_SECONDS, STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, STREAMING_BACKLOG_FILES
Example: RUN_DURATION_SECONDS

opstring: Values: GREATER_THAN
Example: GREATER_THAN

valueint64: Specifies the threshold value that the health metric should obey to satisfy the health rule.
Example: 10

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes

quartz_cron_expressionstring: A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.
Example: 20 30 * * * ?

timezone_idstring: A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.
Example: Europe/London

pause_statusstring: Indicate whether this schedule is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

triggerobject

Show child attributesHide child attributes

pause_statusstring: Whether this trigger is paused or not.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

file_arrivalobjectRequired

File arrival trigger settings.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

periodicobjectRequired

Periodic trigger settings.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

table_updateobjectRequired

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes

pause_statusstring: Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.
Default: UNPAUSED
Values: UNPAUSED, PAUSED

task_retry_modestring: Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.
Values: NEVER, ON_FAILURE

max_concurrent_runsint32: An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.
Default: 1
Example: 10

tasksarray of object

Show child attributesHide child attributes

task_keystring: A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

depends_onarray of object

Show child attributesHide child attributes

task_keystring: The name of the task this task depends on.
Example: Task_Key
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

outcomestring: Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

ALL_SUCCESS: All dependencies have executed and succeeded
AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
NONE_FAILED: None of the dependencies have failed and at least one was executed
ALL_DONE: All dependencies have been completed
AT_LEAST_ONE_FAILED: At least one dependency failed
ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values: ALL_SUCCESS, ALL_DONE, NONE_FAILED, AT_LEAST_ONE_SUCCESS, ALL_FAILED, AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32: An optional timeout applied to each run of this job task. A value of 0 means no timeout.
Default: 0
Example: 86400

healthobject

Show child attributesHide child attributes

rulesarray of object

email_notificationsobject

An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

Show child attributesHide child attributes

on_startarray of string: A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_successarray of string: A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_failurearray of string: A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.
Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string: A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.
Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview: A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.
Example: user.name@databricks.com

no_alert_for_skipped_runsboolean: If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.
Default: false
Example: false

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

Show child attributesHide child attributes

no_alert_for_skipped_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is skipped.
Default: false
Example: false

no_alert_for_canceled_runsboolean: If true, do not send notifications to recipients specified in on_failure if the run is canceled.
Default: false
Example: false

alert_on_last_attemptboolean: If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.
Default: false
Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

Show child attributesHide child attributes

on_startarray of object: An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

on_successarray of object: An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

on_failurearray of object: An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

on_duration_warning_threshold_exceededarray of object: An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

on_streaming_backlog_exceededarray of objectPublic Preview: An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

descriptionstring: An optional description for this task.
Example: This is the description for this task.
Constraints: <= 1000 characters

environment_keystringRequired: The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

disabledboolean: An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.
Default: false

computeobjectBeta

Task level compute configuration.

Show child attributesHide child attributes

hardware_acceleratorstringBeta: Hardware accelerator configuration for Serverless GPU workloads.
Values: GPU_1X_A10, GPU_8X_H100

notebook_taskobjectRequired

The task runs a notebook when the notebook_task field is present.

Show child attributesHide child attributes

notebook_pathstring: The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.
Example: /Users/user.name@databricks.com/notebook_to_run

base_parametersobject

If the notebook takes a parameter that is not specified in the job’s base_parameters or the run-now override parameters, the default value from the notebook is used.

Retrieve these parameters in a notebook using dbutils.widgets.get.

The JSON representation of this field cannot exceed 1MB.

sourcestring

WORKSPACE: Notebook is located in <Databricks> workspace.
GIT: Notebook is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

warehouse_idstring

Optional warehouse_id to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.

Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.

Example: ab12cd34efgh567i

spark_jar_taskobjectRequired

The task runs a JAR when the spark_jar_task field is present.

Show child attributesHide child attributes

jar_uristring

See the examples of classic and serverless compute usage at the top of the page.

main_class_namestring

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.

The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

Example: com.databricks.ComputeModels

parametersarray of string

Parameters passed to the main method.

Use Task parameter variables to set parameters containing information about job runs.

run_as_replboolean: Deprecated. A value of false is no longer supported.
Default: true

spark_python_taskobjectRequired

The task runs a Python file when the spark_python_task field is present.

Show child attributesHide child attributes

python_filestring: The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with /. For files stored in a remote repository, the path must be relative. This field is required.
Example: dbfs:/path/to/file.py

parametersarray of string

Command line parameters passed to the Python file.

Use Task parameter variables to set parameters containing information about job runs.

sourcestring

WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.
GIT: The Python file is located in a remote Git repository.

Values: WORKSPACE, GIT

Example: WORKSPACE

spark_submit_taskobjectRequired

(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

Show child attributesHide child attributes

parametersarray of string

Command-line parameters passed to spark submit.

Use Task parameter variables to set parameters containing information about job runs.

pipeline_taskobjectRequired

The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

Show child attributesHide child attributes

pipeline_idstring: The full name of the pipeline task to execute.
Example: a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5

parametersobjectBeta: Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

python_wheel_taskobjectRequired

The task runs a Python wheel when the python_wheel_task field is present.

Show child attributesHide child attributes

package_namestring: Name of the package to execute

entry_pointstring: Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using $packageName.$entryPoint()

parametersarray of string: Command-line parameters passed to Python wheel task. Leave it empty if named_parameters is not null.

named_parametersobject: Command-line parameters passed to Python wheel task in the form of ["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty if parameters is not null.

dbt_taskobjectRequired

The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

Show child attributesHide child attributes

project_directorystring: Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.

commandsarray of string: A list of dbt commands to execute. All commands must start with dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.

schemastring: Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the default schema is used.

warehouse_idstring: ID of the SQL warehouse to connect to. If provided, we automatically generate and provide the profile and connection details to dbt. It can be overridden on a per-command basis by using the --profiles-dir command line argument.
Example: 30dade0507d960d1

profiles_directorystring: Optional (relative) path to the profiles directory. Can only be specified if no warehouse_id is specified. If no warehouse_id is specified and this folder is unset, the root directory is used.

catalogstring: Optional name of the catalog to use. The value is the top level in the 3-level namespace of Unity Catalog (catalog / schema / relation). The catalog value can only be specified if a warehouse_id is specified. Requires dbt-databricks >= 1.1.1.
Example: main

sourcestring

WORKSPACE: Project is located in <Databricks> workspace.
GIT: Project is located in cloud Git provider.

Values: WORKSPACE, GIT

Example: WORKSPACE

sql_taskobjectRequired

The task runs a SQL query or file, or it refreshes a SQL alert or a legacy SQL dashboard when the sql_task field is present.

Show child attributesHide child attributes

parametersobject: Parameters to be used for each run of this job. The SQL alert task does not support custom parameters.

queryobjectRequired: If query, indicates that this job must execute a SQL query.

dashboardobjectRequired: If dashboard, indicates that this job must refresh a SQL dashboard.

alertobjectRequired: If alert, indicates that this job must refresh a SQL alert.

fileobjectRequired: If file, indicates that this job runs a SQL file in a remote Git repository.

warehouse_idstring: The canonical identifier of the SQL warehouse. Recommended to use with serverless or pro SQL warehouses. Classic SQL warehouses are only supported for SQL alert, dashboard and query tasks and are limited to scheduled single-task jobs.

run_job_taskobjectRequired

The task triggers another job when the run_job_task field is present.

Show child attributesHide child attributes

job_idint64: ID of the job to trigger.

job_parametersobject: Job-level parameters used to trigger the job.

pipeline_paramsobject: Controls whether the pipeline should perform a full refresh

condition_taskobjectRequired

Show child attributesHide child attributes

opstring

EQUAL_TO, NOT_EQUAL operators perform string comparison of their operands. This means that “12.0” == “12” will evaluate to false.
GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL operators perform numeric comparison of their operands. “12.0” >= “12” will evaluate to true, “10.0” >= “12” will evaluate to false.

Values: EQUAL_TO, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL, NOT_EQUAL

leftstring: The left operand of the condition task. Can be either a string value or a job state or parameter reference.

rightstring: The right operand of the condition task. Can be either a string value or a job state or parameter reference.

outcomestring: The condition expression evaluation result. Filled in if the task was successfully completed. Can be "true" or "false"

for_each_taskobjectRequired

The task executes a nested task for every input provided when the for_each_task field is present.

Show child attributesHide child attributes

inputsstring: Array for task to iterate on. This can be a JSON string or a reference to an array parameter.
Constraints: <= 5000 characters

concurrencyint32: An optional maximum allowed number of concurrent runs of the task. Set this value if you want to be able to execute multiple runs of the task concurrently.
Example: 20
Constraints: [ 1 .. 100 ]

taskobject: Configuration for the task that will be run for each element in the array

clean_rooms_notebook_taskobjectRequired

The task runs a clean rooms notebook when the clean_rooms_notebook_task field is present.

Show child attributesHide child attributes

clean_room_namestring: The clean room that the notebook belongs to.

notebook_namestring: Name of the notebook being run.

etagstring: Checksum to validate the freshness of the notebook resource (i.e. the notebook being run is the latest version). It can be fetched by calling the cleanroomassets/get API.

notebook_base_parametersobject: Base parameters to be used for the clean room notebook job.

alert_taskobjectRequiredPublic Preview

The task evaluates a <Databricks> alert and sends notifications to subscribers when the alert_task field is present.

Show child attributesHide child attributes

alert_idstringPublic Preview: The alert_id is the canonical identifier of the alert.

warehouse_idstringPublic Preview: The warehouse_id identifies the warehouse settings used by the alert task.

workspace_pathstringPublic Preview

The workspace_path is the path to the alert file in the workspace. The path:

must start with "/Workspace"
must be a normalized path. User has to select only one of alert_id or workspace_path to identify the alert.

Example: /Workspace/Users/user@company.com/example.dbalert.json

subscribersarray of objectPublic Preview: The subscribers receive alert evaluation result notifications after the alert task is completed. The number of subscriptions is limited to 100.

power_bi_taskobjectRequiredPublic Preview

The task triggers a Power BI semantic model update when the power_bi_task field is present.

Show child attributesHide child attributes

tablesarray of objectPublic Preview: The tables to be exported to Power BI

warehouse_idstringPublic Preview: The SQL warehouse ID to use as the Power BI data source

power_bi_modelobjectPublic Preview: The semantic model to update

connection_resource_namestringPublic Preview: The resource name of the UC connection to authenticate from <Databricks> to Power BI

refresh_after_updatebooleanPublic Preview: Whether the model should be refreshed after the update

dashboard_taskobjectRequired

The task refreshes a dashboard and sends a snapshot to subscribers.

Show child attributesHide child attributes

subscriptionobject: Optional: subscription configuration for sending the dashboard snapshot.

warehouse_idstring: Optional: The warehouse id to execute the dashboard with for the schedule. If not specified, the default warehouse of the dashboard will be used.
Example: 47bb1c472649e711

dashboard_idstring: The identifier of the dashboard to refresh.
Example: 01ef0cb45e2a1da4a61950e9b8789ce9

ai_runtime_taskobjectRequiredPublic Preview

The task runs a multi-gpu compute workload on Databricks AI Runtime. Specify the accelerator type and count, the command to run, and where the workload's code and MLflow output are stored.

Show child attributesHide child attributes

experimentstringPublic Preview: MLflow experiment name for this run. If an experiment with this name already exists under the calling user, the run is appended to it; otherwise a new experiment is created. To target a specific MLflow storage location (for example, when running as a service principal), set mlflow_experiment_directory.
Example: llama-fine-tune

deploymentsarray of objectPublic Preview: Deployment specs for this task. Exactly one deployment is currently supported (a single entry where every node runs the same command); this is a current-Preview constraint. Role-split workloads (driver + worker, parameter server, separate eval node, etc.) with multiple entries are the eventual intent but not yet supported.

mlflow_runstringPublic Preview: Optional display name for the MLflow run created under experiment. If omitted, MLflow generates a default name.
Example: llama-fine-tune-lora-r16

mlflow_experiment_directorystringPublic Preview: Optional workspace directory under which the MLflow experiment named in experiment is created. Must start with /Workspace. Set this when running as a service principal that has no default user directory; for regular users the experiment defaults to the user's home directory.
Example: /Workspace/Users/alice@databricks.com/experiments

docker_image_urlstringBeta: Optional Docker image URL for a custom container image. When set, the task runs on the specified container image instead of the default <Databricks> client image. Format: {organization}/{repository}:{tag}
Example: nvidia/cuda:13.3.0-cudnn-devel-ubuntu24.04

existing_cluster_idstringRequired: If existing_cluster_id, the ID of an existing cluster that is used for all runs. When running jobs or tasks on an existing cluster, you may need to manually restart the cluster if it stops responding. We suggest running jobs and tasks on new clusters for greater reliability
Example: 0923-164208-meows279

new_clusterobjectRequired

If new_cluster, a description of a new cluster that is created for each run.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

job_cluster_keystringRequired: If job_cluster_key, this task is executed reusing the cluster specified in job.settings.job_clusters.
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

librariesarray of object

An optional list of libraries to be installed on the cluster. The default value is an empty list.

Show child attributesHide child attributes

jarstringRequired: URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "s3://my-bucket/library.jar" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the JAR library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "abfss://my-bucket/library.jar" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the jar library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "jar": "/Workspace/path/to/library.jar" }, { "jar" : "/Volumes/path/to/library.jar" } or { "jar": "gs://my-bucket/library.jar" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

eggstringRequired: Deprecated. URI of the egg library to install. Installing Python egg files is deprecated and is not supported in Databricks Runtime 14.0 and above.

pypiobjectRequired: Specification of a PyPi library to be installed. For example: { "package": "simplejson" }

mavenobjectRequired: Specification of a maven library to be installed. For example: { "coordinates": "org.jsoup:jsoup:1.7.2" }

cranobjectRequired: Specification of a CRAN library to be installed as part of the library

whlstringRequired: URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and S3 URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "s3://my-bucket/library.whl" }. If S3 is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM role to access the S3 URI.
Azure
URI of the wheel library to install. Supported URIs include Workspace paths, Unity Catalog Volumes paths, and ADLS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "abfss://my-bucket/library.whl" }. If ADLS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with a Microsoft Entra ID service principal to access the ADLS URI.
GCP
URI of the wheel library to be installed. Supported URIs include Workspace path, UC Volumes path, and GCS URIs. For example: { "whl": "/Workspace/path/to/library.whl" }, { "whl" : "/Volumes/path/to/library.whl" } or { "whl": "gs://my-bucket/library.egg" }. If GCS is used, please make sure the cluster has read access on the library. You may need to launch the cluster with an IAM service account to access the GCS URI.

requirementsstringRequired: URI of the requirements.txt file to install. Only Workspace paths and Unity Catalog Volumes paths are supported. For example: { "requirements": "/Workspace/path/to/requirements.txt" } or { "requirements" : "/Volumes/path/to/requirements.txt" }

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

job_clustersarray of object

A list of job cluster specifications that can be shared and reused by tasks of this job. Libraries cannot be declared in a shared job cluster. You must declare dependent libraries in task settings.

Constraints: <= 100 items

Show child attributesHide child attributes

job_cluster_keystring: A unique name for the job cluster. This field is required and must be unique within the job. JobTaskSettings may refer to this field to determine which cluster to launch for the task execution.
Example: auto_scaling_cluster
Constraints: [ 1 .. 100 ] characters, ^[\\w\\-\\_]+$

new_clusterobject

If new_cluster, a description of a cluster that is created for each task.

Show child attributesHide child attributes

apply_policy_default_valuesboolean: Default: false

cluster_namestring: Cluster name requested by the user. This doesn't have to be unique. If not specified at creation, the cluster name will be an empty string. For job clusters, the cluster name is automatically set based on the job and job run IDs.

spark_versionstring: The Spark version of the cluster, e.g. 3.3.x-scala2.11. A list of available Spark versions can be retrieved by using the clusters/sparkVersions API call.

spark_confobject: An object containing a set of optional, user-specified Spark configuration key-value pairs. Users can also pass in a string of extra JVM options to the driver and the executors via spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

aws_attributesobject: Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

azure_attributesobject: Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

gcp_attributesobject: Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

node_type_idstring: This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

worker_node_type_flexibilityobject: Flexible node type configuration for worker nodes.

driver_node_type_flexibilityobject: Flexible node type configuration for the driver node.

ssh_public_keysarray of string: SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

Currently, <Databricks> allows at most 45 custom tags
Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject: The configuration for delivering spark logs to a long-term storage destination. Three kinds of destinations (DBFS, S3 and Unity Catalog volumes) are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

spark_env_varsobject

autotermination_minutesint32: Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. Users can also set this value to 0 to explicitly disable automatic termination.

enable_elastic_diskboolean: Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space.
AWS
Autoscaling Local Storage: when enabled, this cluster will dynamically acquire additional disk space when its Spark workers are running low on disk space. This feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.

init_scriptsarray of object: The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

docker_imageobject: Custom docker image BYOC

instance_pool_idstring: The optional ID of the instance pool to which the cluster belongs.

single_user_namestring: Single user name if data_security_mode is SINGLE_USER

policy_idstring: The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean: Whether to enable LUKS on cluster VMs' local disks

driver_instance_pool_idstring: The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

workload_typeobject

data_security_modestring: Values: NONE, SINGLE_USER, USER_ISOLATION, LEGACY_TABLE_ACL, LEGACY_PASSTHROUGH, LEGACY_SINGLE_USER, LEGACY_SINGLE_USER_STANDARD, DATA_SECURITY_MODE_STANDARD, DATA_SECURITY_MODE_DEDICATED, DATA_SECURITY_MODE_AUTO

runtime_enginestring

Determines the cluster's runtime engine, either standard or Photon.

This field is not compatible with legacy spark_version values that contain -photon-. Remove -photon- from the spark_version and set runtime_engine to PHOTON.

If left unspecified, the runtime engine defaults to standard unless the spark_version contains -photon-, in which case Photon will be used.

Values: NULL, STANDARD, PHOTON

kindstring: Values: COMPUTE_KIND_UNSPECIFIED, CLASSIC_PREVIEW

use_ml_runtimeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

effective_spark_version is determined by spark_version (DBR release), this field use_ml_runtime, and whether node_type_id is gpu node or not.

is_single_nodeboolean

This field can only be used when kind = CLASSIC_PREVIEW.

When set to true, <Databricks> will automatically set single node related custom_tags, spark_conf, and num_workers

remote_disk_throughputint32: If set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED disks.

total_initial_remote_disk_sizeint32: If set, what the total initial volume size (in GB) of the remote disks should be. Supported for GCP.

dependency_modestringBeta: Controls dependency configuration for the cluster.
Values: DEPENDENCY_MODE_UNSPECIFIED, DEPENDENCY_MODE_ENVIRONMENTS, DEPENDENCY_MODE_CLUSTER_LIBRARIES, DEPENDENCY_MODE_AUTO

num_workersint32Required

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

autoscaleobjectRequired: Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

git_sourceobject

An optional specification for a remote Git repository containing the source code used by tasks. Version-controlled source code is supported by notebook, dbt, Python script, and SQL File tasks.

If git_source is set, these tasks retrieve the file from the remote repository by default. However, this behavior can be overridden by setting source to WORKSPACE on the task.

Note: dbt and SQL File tasks support only version-controlled sources. If dbt or SQL File tasks are used, git_source must be defined on the job.

Show child attributesHide child attributes

git_urlstring: URL of the repository to be cloned by this job.
Example: https://github.com/databricks/databricks-cli
Constraints: <= 300 characters

git_providerstring: Unique identifier of the service used to host the Git repository. The value is case insensitive.

git_branchstringRequired: Name of the branch to be checked out and used by this job. This field cannot be specified in conjunction with git_tag or git_commit.
Example: main
Constraints: <= 255 characters

git_tagstringRequired: Name of the tag to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_commit.
Example: release-1.0.0
Constraints: <= 255 characters

git_commitstringRequired: Commit to be checked out and used by this job. This field cannot be specified in conjunction with git_branch or git_tag.
Example: e0056d01
Constraints: <= 64 characters

git_snapshotobject

Show child attributesHide child attributes

used_commitstring: Commit that was used to execute the run. If git_branch was specified, this points to the HEAD of the branch at the time of the run; if git_tag was specified, this points to the commit the tag points to.
Example: 4506fdf41e9fa98090570a34df7a5bce163ff15f

sparse_checkoutobject

Show child attributesHide child attributes

patternsarray of string: List of patterns to include for sparse checkout.

tagsobject: A map of tags associated with the job. These are forwarded to the cluster as cluster tags for jobs clusters, and are subject to the same limitations as cluster tags. A maximum of 25 tags can be added to the job.

formatstring: Used to tell what is the format of the job. This field is ignored in Create/Update/Reset calls. When using the Jobs API 2.1 this value is always set to "MULTI_TASK".
Values: SINGLE_TASK, MULTI_TASK
Example: MULTI_TASK

queueobject

The queue settings of the job.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

parametersarray of object

Job-level parameter definitions

Show child attributesHide child attributes

namestring: The name of the defined parameter. May only contain alphanumeric characters, _, -, and .
Example: table
Constraints: ^[\\w\\-.]+$

defaultstring: Default value of the parameter.
Example: users

run_asobject

Show child attributesHide child attributes

user_namestringRequired: The email of an active workspace user. Non-admin users can only set this field to their own email.
Example: user@databricks.com

service_principal_namestringRequired: Application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
Example: 692bc6d0-ffa3-11ed-be56-0242ac120002

edit_modestring

Edit mode of the job.

UI_LOCKED: The job is in a locked UI state and cannot be modified.
EDITABLE: The job is in an editable state and can be modified.

Values: UI_LOCKED, EDITABLE

deploymentobject

Deployment information for jobs managed by external sources.

Show child attributesHide child attributes

kindstring

The kind of deployment that manages the job.

BUNDLE: The job is managed by Databricks Asset Bundle.
SYSTEM_MANAGED: The job is managed by <Databricks> and is read-only.

Values: BUNDLE, SYSTEM_MANAGED

metadata_file_pathstring: Path of the file that contains deployment metadata.

environmentsarray of object

Constraints: <= 10 items

Show child attributesHide child attributes

environment_keystring: The key of an environment. It has to be unique within a job.

specobject

Show child attributesHide child attributes

clientstring: Use environment_version instead.
Example: 1

dependenciesarray of string: List of pip dependencies, as supported by the version of pip in this environment. Each dependency is a valid pip requirements file line per https://pip.pypa.io/en/stable/reference/requirements-file-format/. Allowed dependencies include a requirement specifier, an archive URL, a local project path (such as WSFS or UC Volumes in <Databricks>), or a VCS project URL.

base_environmentstring: The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
AWS
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see https://docs.databricks.com/aws/en/compute/serverless/dependencies#base-environments-for-job-tasks.
Azure
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://learn.microsoft.com/azure/databricks/compute/serverless/dependencies#base-environments-for-job-tasks.
GCP
The base environment this environment is built on top of. A base environment defines the environment version and a list of dependencies for serverless compute. The value can be a file path to a custom env.yaml file (e.g., /Workspace/path/to/env.yaml). Support for a <Databricks>-provided base environment ID (e.g., workspace-base-environments/databricks_ai_v4) and workspace base environment ID (e.g., workspace-base-environments/dbe_b849b66e-b31a-4cb5-b161-1f2b10877fb7) is in Beta. Either environment_version or base_environment can be provided. For more information about <Databricks>-provided base environments, see the list workspace base environments API. For more information, see
https://docs.databricks.com/gcp/en/compute/serverless/dependencies#base-environments-for-job-tasks.

environment_versionstring: Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
AWS
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer. See https://docs.databricks.com/aws/release-notes/serverless/#serverless-environment-versions.
Azure
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://learn.microsoft.com/azure/databricks/release-notes/serverless/#serverless-environment-versions.
GCP
Either environment_version or base_environment needs to be provided. Environment version used by the environment. Each version comes with a specific Python version and a set of Python packages. The version is a string, consisting of an integer.
See https://docs.databricks.com/gcp/release-notes/serverless/#serverless-environment-versions.
Example: 5

java_dependenciesarray of string: List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
AWS
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar. See https://docs.databricks.com/aws/en/jobs/jar.
Azure
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://learn.microsoft.com/en-gb/azure/databricks/jobs/jar.
GCP
List of java dependencies. Each dependency is a string representing a java library path. For example: /Volumes/path/to/test.jar.
See https://docs.databricks.com/gcp/en/jobs/jar.

budget_policy_idstringPublic Preview: The id of the user specified budget policy to use for this job. If not specified, a default budget policy may be applied when creating or modifying the job. See effective_budget_policy_id for the budget policy used by this workload.
Example: 550e8400-e29b-41d4-a716-446655440000

performance_targetstring

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Default: PERFORMANCE_OPTIMIZED

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

triggersarray of objectBeta

Show child attributesHide child attributes

pause_statusstringBeta: Whether this trigger is paused. Defaults to UNPAUSED when unset; the server always returns an explicit value on read.
Values: UNPAUSED, PAUSED

periodicobjectBeta

Trigger type: exactly one must be set; mutual exclusivity is enforced in the API handler Periodic trigger configuration.

Show child attributesHide child attributes

intervalint32: The interval at which the trigger should run.

unitstring: The unit of time for the interval.
Values: TIME_UNIT_UNSPECIFIED, HOURS, DAYS, WEEKS, MINUTES

scheduleobjectBeta

Cron schedule trigger configuration.

Show child attributesHide child attributes

quartz_cron_expressionstringBeta: A Cron expression using Quartz syntax that describes the schedule for this trigger. See Cron Trigger for details.
Example: 20 30 * * * ?

timezone_idstringBeta: A Java timezone ID. The schedule is resolved with respect to this timezone. See Java TimeZone for details.
Example: Europe/London

continuousobjectBeta

Continuous trigger configuration.

Show child attributesHide child attributes

task_retry_modestringBeta: Whether the continuous job applies task-level retries. Defaults to NEVER.
Values: NEVER, ON_FAILURE

file_arrivalobjectBeta

File arrival trigger configuration.

Show child attributesHide child attributes

urlstring: URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

table_updateobjectBeta

Table update trigger configuration.

Show child attributesHide child attributes

table_namesarray of string: A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32: If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32: If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring: The table(s) condition based on which to trigger a job run.
Values: ANY_UPDATED, ALL_UPDATED
Example: ALL_UPDATED

max_retriesint32: An optional maximum number of times to retry an unsuccessful run. A run is considered to be unsuccessful if it completes with the FAILED result_state or INTERNAL_ERROR life_cycle_state. The value -1 means to retry indefinitely and the value 0 means to never retry.
Default: 0
Example: 10

min_retry_interval_millisint32: An optional minimal interval in milliseconds between the start of the failed run and the subsequent retry run. The default behavior is that unsuccessful runs are immediately retried.
Example: 2000

retry_on_timeoutboolean: An optional policy to specify whether to retry a job when it times out. The default behavior is to not retry on timeout.
Default: false
Example: true

disable_auto_optimizationboolean: An option to disable auto optimization in serverless
Default: false
Example: true

fields_to_removearray of string: Remove top-level fields in the job settings. Removing nested fields is not supported, except for tasks and job clusters (tasks/task_1). This field is optional.

View as Markdown

Request

{
  "job_id": 11223344,
  "new_settings": {
    "name": "A multitask job",
    "description": "This job contain multiple tasks that are required to produce the weekly shark sightings report.",
    "email_notifications": {},
    "webhook_notifications": {},
    "notification_settings": {},
    "timeout_seconds": 86400,
    "health": {},
    "schedule": {},
    "trigger": {},
    "continuous": {},
    "max_concurrent_runs": 10,
    "tasks": [
      {
        "depends_on": [],
        "description": "Extracts session data from events",
        "existing_cluster_id": "0923-164208-meows279",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/Sessionize.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.Sessionize",
          "parameters": [
            "--data",
            "dbfs:/path/to/data.json"
          ]
        },
        "task_key": "Sessionize",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [],
        "description": "Ingests order data",
        "job_cluster_key": "auto_scaling_cluster",
        "libraries": [
          {
            "jar": "dbfs:/mnt/databricks/OrderIngest.jar"
          }
        ],
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "retry_on_timeout": false,
        "spark_jar_task": {
          "main_class_name": "com.databricks.OrdersIngest",
          "parameters": [
            "--data",
            "dbfs:/path/to/order-data.json"
          ]
        },
        "task_key": "Orders_Ingest",
        "timeout_seconds": 86400
      },
      {
        "depends_on": [
          {
            "task_key": "Orders_Ingest"
          },
          {
            "task_key": "Sessionize"
          }
        ],
        "description": "Matches orders with user sessions",
        "max_retries": 3,
        "min_retry_interval_millis": 2000,
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        },
        "notebook_task": {
          "base_parameters": {
            "age": "35",
            "name": "John Doe"
          },
          "notebook_path": "/Users/user.name@databricks.com/Match"
        },
        "retry_on_timeout": false,
        "run_if": "ALL_SUCCESS",
        "task_key": "Match",
        "timeout_seconds": 86400
      }
    ],
    "job_clusters": [
      {
        "job_cluster_key": "auto_scaling_cluster",
        "new_cluster": {
          "autoscale": {
            "max_workers": 16,
            "min_workers": 2
          },
          "node_type_id": null,
          "spark_conf": {
            "spark.speculation": true
          },
          "spark_version": "7.3.x-scala2.12"
        }
      }
    ],
    "git_source": {},
    "tags": {
      "cost-center": "engineering",
      "team": "jobs"
    },
    "format": "MULTI_TASK",
    "queue": {},
    "parameters": [
      {}
    ],
    "run_as": {},
    "edit_mode": "string",
    "deployment": {},
    "environments": [
      {}
    ],
    "budget_policy_id": "550e8400-e29b-41d4-a716-446655440000",
    "performance_target": "string",
    "triggers": [
      {}
    ],
    "max_retries": 10,
    "min_retry_interval_millis": 2000,
    "retry_on_timeout": true,
    "disable_auto_optimization": true
  },
  "fields_to_remove": [
    "libraries",
    "schedule",
    "tasks/task_1",
    "job_clusters/Default"
  ]
}

Response

{}

Run Now GA

POST /api/2.2/jobs/run-now

Run a job and return the run_id of the triggered run.

API scopes: jobs

Request body

job_idint64: The ID of the job to be executed
Example: 11223344

job_parametersobject: Job-level parameters used in the run. for example "param": "overriding_val"

idempotency_tokenstring

An optional token to guarantee the idempotency of job run requests. If a run with the provided token already exists, the request does not create a new run but returns the ID of the existing run instead. If a run with the provided token is deleted, an error is returned.

If you specify the idempotency token, upon failure you can retry until the request succeeds. <Databricks> guarantees that exactly one run is launched with that idempotency token.

This token must have at most 64 characters.

Example: 8f018174-4792-40d5-bcbc-3e6a527352c8

queueobject

The queue settings of the run.

Show child attributesHide child attributes

enabledboolean: If true, enable queueing for the job. This is a required field.
Default: true
Example: true

onlyarray of string

A list of task keys to run inside of the job. If this field is not provided, all tasks in the job will be run.

Prefix a task key with + to also run its upstream tasks, or suffix it with + to also run its downstream tasks. For example, +my_task runs my_task and everything upstream of it, my_task+ runs my_task and everything downstream of it, and +my_task+ runs both. A task key with no + runs only that task.

Example: notebook_task_1

performance_targetstring

The performance mode on a serverless job. The performance target determines the level of compute performance or cost-efficiency for the run. This field overrides the performance target defined on the job level.

STANDARD: Enables cost-efficient execution of serverless workloads.
PERFORMANCE_OPTIMIZED: Prioritizes fast startup and execution times through rapid scaling and optimized cluster performance.

Values: PERFORMANCE_TARGET_UNSPECIFIED, PERFORMANCE_OPTIMIZED, STANDARD

pipeline_paramsobject

Controls whether the pipeline should perform a full refresh

Show child attributesHide child attributes

full_refreshboolean: If true, triggers a full refresh on the spark declarative pipeline.
Default: false

refresh_selectionarray of stringBeta: A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta: A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta: A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta: Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

Response

run_idint64: The globally unique ID of the newly triggered run.
Example: 455644833

number_in_jobint64: A unique identifier for this job run. This is set to the same value as run_id.
Example: 455644833

View as Markdown

Request

{
  "job_id": 11223344,
  "job_parameters": {},
  "idempotency_token": "8f018174-4792-40d5-bcbc-3e6a527352c8",
  "queue": {
    "enabled": true
  },
  "only": "notebook_task_1",
  "performance_target": "string",
  "pipeline_params": {
    "full_refresh": true,
    "refresh_selection": [
      "string"
    ],
    "full_refresh_selection": [
      "string"
    ],
    "reset_checkpoint_selection": [
      "string"
    ],
    "refresh_flow_selection": [
      "string"
    ]
  }
}

Response

{
  "run_id": 455644833,
  "number_in_job": 455644833
}

Get GA​

Parameters​

Response​

List GA​

Parameters​

Response​

Create GA​

Request body​

Plain Python:

Multi-GPU via accelerate:

Distributed via torchrun:

Response​

Update GA​

Request body​

Delete GA​

Request body​

Update Job GA​

Request body​

Run Now GA​

Request body​

Response​

Get GA

Parameters

Response

List GA

Parameters

Response

Create GA

Request body

Response

Update GA

Request body

Delete GA

Request body

Update Job GA

Request body

Run Now GA

Request body

Response