Production job scheduling cheat sheet

This article aims to provide clear and opinionated guidance for production job scheduling. Using best practices can help reduce costs, improve performance, and tighten security.

Best Practice

Impact

Docs

Use jobs clusters for automated workflows

Cost: Jobs clusters are billed at lower rates than interactive clusters.

Restart long-running clusters

Security: Restart clusters to take advantage of patches and bug fixes to the Databricks Runtime.

Use service principals instead of user accounts to run production jobs

Security: If jobs are owned by individual users, when those users leave the org, these jobs may stop running.

Use Databricks Jobs for orchestration whenever possible

Cost: There’s no need to use external tools to orchestrate if you are only orchestrating workloads on Databricks.

Use latest LTS version of Databricks Runtime

Performance and cost: Databricks is always improving Databricks Runtime for usability, performance, and security.

Don’t store production data in DBFS root

Security: When data is stored in the DBFS root, all users can access it.