Run tasks conditionally in a Databricks job
By default, a job task runs when its dependencies have run and have all succeeded, but you can also configure tasks in a Databricks job to only run when specific conditions are met. For example, you can use the Run if condition to run a task even when some or all of its dependencies have failed, allowing your job to recover from failures and continue running.
Change the Run if
condition of a task
You can configure a Run if
condition when you edit a task with one or more dependencies. To add the condition to the task, select the condition from the Run if dropdown menu in the task configuration. The Run if
condition is evaluated after all task dependencies have been completed. You can also add a Run if
condition when you add a new task with one or more dependencies.
Run if
condition options
You can add the following Run if
conditions to a task:
All succeeded: All dependencies have run and succeeded. This is the default condition to run a task. The task is marked as
Upstream failed
if the condition is not met.At least one succeeded: At least one dependency has succeeded. The task is marked as
Upstream failed
if the condition is not met.None failed: None of the dependencies failed, and at least one dependency was run. The task is marked as
Upstream failed
if the condition is not met.All done: The task is run after all its dependencies have run, regardless of the status of the dependent runs. This condition allows you to define a task that is run without depending on the outcome of its dependent tasks.
At least one failed: At least one dependency failed. The task is marked as
Excluded
if the condition is not met.All failed: All dependencies have failed. The task is marked as
Excluded
if the condition is not met.
Note
Tasks configured to handle failures are marked as
Excluded
if theirRun if
condition is not met. Excluded tasks are skipped and are treated as successful.If all task dependencies are excluded, the task is also excluded, regardless of its
Run if
condition.If you cancel a task run, the cancellation propagates through downstream tasks, and tasks with a
Run if
condition that handles failure are run, for example, to ensure a cleanup task runs when a task run is canceled.
How does Databricks Jobs determine job run status?
Databricks Jobs determines whether a job run was successful based on the outcome of the job’s leaf tasks. A leaf task is a task that has no downstream dependencies. A job run can have one of three outcomes:
Succeeded: All tasks were successful.
Succeeded with failures: Some tasks failed, but all leaf tasks were successful.
Failed: One or more leaf tasks failed.