Databricks Runtime Versions

Databricks Runtime is the set of core components that run on the clusters managed by Databricks. It includes Apache Spark but also adds a number of components and updates that substantially improve the usability, performance, and security of big data analytics.

You can choose from among many supported runtime versions when you create a cluster.

../../_images/runtime-version.png

Runtime components

Databricks Runtime consists of the following components:

  • Apache Spark: each runtime version contains a specific Apache Spark version
  • Databricks IO: a layer on top of Apache Spark that provides additional reliability and blazing performance
  • Databricks Serverless: a layer on top of Apache Spark that provides fine-grained resource sharing to optimize cloud costs
  • Ubuntu and its accompanying system libraries
  • Pre-installed Java, Scala, Python, and R languages
  • Pre-installed Java, Scala, Python, and R libraries
  • GPU libraries for GPU-enabled clusters
  • Databricks services that integrate with other components of the platform, such as notebooks, jobs, and cluster manager

The Databricks Runtime Release Notes list the library versions included in each Runtime version.

Versioning

We release new versions of Databricks Runtime on a regular basis.

  • Major Releases are represented by an increment to the version number that precedes the decimal point (the jump from 3.5 to 4.0, for example). They are released when there are major changes, some of which may not be backwards-compatible.
  • Feature Releases are represented by an increment to the version number that follows the decimal point (the jump from 3.4 to 3.5, for example). Each major release includes multiple feature releases. Feature releases are always backwards compatible with previous releases within their major release.
  • Long Term Support releases are represented by an “-LTS” suffix (for example, 3.5-LTS) and are the “canonical” feature version of the major release, for which we provide two full years of support. We recommend these releases for your production jobs. See the deprecation policy described below for more information.

Phases of a version

Phase Guarantees
Beta Support SLAs are not applicable.
Full Support Major stability and security fixes are backported. New features will be available in the next version.
Marked for deprecation Version will be deprecated (unsupported) soon. This phase will last no less than 3 months, although the actual duration will vary depending on version type. See Deprecation policy for more information.
Deprecated

Version is unsupported:

  • Not available for selection in the UI
  • Workloads running on these versions receive no Databricks support
  • Databricks will not backport fixes
  • Users can continue to use the API to create clusters
Sunset Typically, Databricks removes a version completely from the API only when its usage drops to 0. Your scheduled workloads are therefore guaranteed to run properly regardless of the deprecation schedule. If we make exceptions to this rule, we will give ample notice.

Deprecation policy

The Databricks deprecation policy depends on version type.

Feature version

We give 4 months notice before deprecating a feature version. After 4 months from the deprecation announcement, we remove the version from the Create Cluster and Edit Cluster pages. We do not backport fixes and we provide no support for those versions. You can continue to create clusters from the API if you have automated jobs running.

Long Term Support (LTS) version

For every major version, we identify a pinned version for which we offer two years of support from the date of release. We recommend LTS releases for your production jobs. After two years, we mark the LTS version for deprecation, deprecating it one year later.

List of releases

Current Releases

Version Spark Version Release Date Deprecation Announcement Deprecation Date
4.3 Spark 2.3 Aug 10, 2018
4.2 Spark 2.3 Jul 09, 2018 Nov 05, 2018 Mar 05, 2019
4.1 Spark 2.3 May 17, 2018 Sep 17, 2018 Jan 17, 2019
4.0 Spark 2.3 Mar 01, 2018 Jul 01, 2018 Nov 01, 2018
3.5-LTS Spark 2.2 Dec 21, 2017 Jan 01, 2019 Jan 01, 2020

Marked for Deprecation

Version Spark Version Release Date Deprecation Announcement Deprecation Date
4.0 Spark 2.3 Mar 01, 2018 Jul 01, 2018 Nov 01, 2018

Deprecated Releases

Version Spark Version Release Date Deprecation Announcement Deprecation Date
3.4 Spark 2.2 Nov 20, 2017 Mar 31, 2018 Jul 30, 2018
3.3 Spark 2.2 Oct 04, 2017 Mar 31, 2018 Jul 30, 2018
3.2 Spark 2.2 Sep 05, 2017 Jan 30, 2018 Apr 30, 2018
3.1 Spark 2.2 Aug 04, 2017 Oct 30, 2017
3.0 Spark 2.2 Jul 11, 2017 Sep 05, 2017
Spark 2.1 (Auto Updating) Spark 2.1 Dec 22, 2016 Mar 31, 2018 Jul 30, 2018
Spark 2.1.1-db6 Spark 2.1 Aug 03, 2017 Mar 31, 2018 Jul 30, 2018
Spark 2.1.1-db5 Spark 2.1 May 31, 2017 Aug 03, 2017
Spark 2.1.1-db4 Spark 2.1 Apr 25, 2017 Mar 31, 2018 Jul 30, 2018
Spark 2.0 (Auto Updating) Spark 2.0 Jul 26, 2016 Jan 30, 2018 Apr 30, 2018
Spark 2.0.2-db4 Spark 2.0 Mar 24, 2017 Jan 30, 2018 Apr 30, 2018
Spark 1.6.3-db2 Spark 1.6 Mar 24, 2017 Jan 30, 2018 Jun 30, 2018

REST API version string

The structure of a Databricks Runtime version string in the REST API is:

For versions 3.x and above:

<M>.<F>.x[-gpu]-scala<scala-version>

where

  • M - Databricks Runtime major release
  • F - Databricks Runtime feature release
  • gpu - GPU-enabled runtime
  • scala-version - version of Scala used to compile Spark: 2.10 or 2.11

For example, 3.5.x-scala2.10 and 4.1.x-gpu-scala2.11. The List of releases tables map Databricks Runtime versions to the Spark version contained in the Runtime.

For versions 2.x and below (deprecated):

<M>.<F>.<m>-db<n>-scala<scala-version>

where

  • M - Apache Spark major release
  • F - Apache Spark feature release
  • m - Apache Spark maintenance update
  • n - Databricks Runtime version
  • scala-version - version of Scala used to compile Spark: 2.10 or 2.11

For example, 2.1.1-db6-scala2.11.