Production job scheduling cheat sheet
This article aims to provide clear and opinionated guidance for production job scheduling. Using best practices can help reduce costs, improve performance, and tighten security.
Best Practice | Impact | Docs |
|---|---|---|
Use serverless compute for jobs | Cost: Serverless jobs require no cluster configuration. Databricks manages provisioning and scaling automatically. | |
Use Lakeflow Jobs for orchestration whenever possible | Cost: There's no need to use external tools to orchestrate if you are only orchestrating workloads on Databricks. | |
Use service principals instead of user accounts to run production jobs | Security: If jobs are owned by individual users, when those users leave the org, these jobs may stop running. | |
For classic compute: use jobs clusters for automated workflows | Cost: Jobs clusters are billed at lower rates than interactive clusters. | |
For classic compute: restart long-running clusters | Security: Restart clusters to take advantage of patches and bug fixes to the Databricks Runtime. | |
For classic compute: use latest LTS version of Databricks Runtime | Performance and cost: Databricks is always improving Databricks Runtime for usability, performance, and security. | |
For classic compute: don't store production data in DBFS root | Security: When data is stored in the DBFS root, all users can access it. |