Run tasks conditionally in a Databricks job

By default, a job task runs when its dependencies have run and have all succeeded, but you can also configure tasks in a Databricks job to only run when specific conditions are met. For example, you can use the Run if condition to run a task even when some or all of its dependencies have failed, allowing your job to recover from failures and continue running.

Change the Run if condition of a task

You can configure a Run if condition when you edit a task with one or more dependencies. To add the condition to the task, select the condition from the Run if dropdown menu in the task configuration. The Run if condition is evaluated after all task dependencies have been completed. You can also add a Run if condition when you add a new task with one or more dependencies.

Run if condition options

You can add the following Run if conditions to a task:

  • All succeeded: All dependencies have run and succeeded. This is the default condition to run a task. The task is marked as Upstream failed if the condition is not met.

  • At least one succeeded: At least one dependency has succeeded. The task is marked as Upstream failed if the condition is not met.

  • None failed: None of the dependencies failed, and at least one dependency was run. The task is marked as Upstream failed if the condition is not met.

  • All done: The task is run after all its dependencies have run, regardless of the status of the dependent runs. This condition allows you to define a task that is run without depending on the outcome of its dependent tasks.

  • At least one failed: At least one dependency failed. The task is marked as Excluded if the condition is not met.

  • All failed: All dependencies have failed. The task is marked as Excluded if the condition is not met.

Note

  • Tasks configured to handle failures are marked as Excluded if their Run if condition is not met. Excluded tasks are skipped and are treated as successful.

  • If all task dependencies are excluded, the task is also excluded, regardless of its Run if condition.

  • If you cancel a task run, the cancellation propagates through downstream tasks, and tasks with a Run if condition that handles failure are run, for example, to ensure a cleanup task runs when a task run is canceled.

How does Databricks Jobs determine job run status?

Databricks Jobs determines whether a job run was successful based on the outcome of the job’s leaf tasks. A leaf task is a task that has no downstream dependencies. A job run can have one of three outcomes:

  • Succeeded: All tasks were successful.

  • Succeeded with failures: Some tasks failed, but all leaf tasks were successful.

  • Failed: One or more leaf tasks failed.