Databricks Light is the Databricks packaging of the open source Apache Spark runtime. It provides a runtime option for jobs that don’t need the advanced performance, reliability, or autoscaling benefits provided by Databricks Runtime. In particular, Databricks Light does not support:
Autopilot features such as autoscaling
Highly concurrent, all-purpose clusters
Notebooks, dashboards, and collaboration features
Connectors to various data sources and BI tools
Databricks Light is a runtime environment for jobs (or “automated workloads”). When you run jobs on Databricks Light clusters, they are subject to lower Jobs Light Compute pricing. You can select Databricks Light only when you create or schedule a JAR, Python, or spark-submit job and attach a cluster to that job; you cannot use Databricks Light to run notebook jobs or interactive workloads.
Databricks Light can be used in the same workspace with clusters running on other Databricks runtimes and pricing tiers. You don’t need to request a separate workspace to get started.
The release schedule of Databricks Light runtime follows the Apache Spark runtime release schedule. Any Databricks Light version is based on a specific version of Apache Spark. See the following release notes for more information: