Databricks Runtime includes Apache Spark but also adds a number of components and updates that substantially improve the usability, performance, and security of big data analytics:
- Delta Lake, a next-generation storage layer built on top of Apache Spark that provides ACID transactions, optimized layouts and indexes, and execution engine improvements for building data pipelines.
- Installed Java, Scala, Python, and R libraries
- Ubuntu and its accompanying system libraries
- GPU libraries for GPU-enabled clusters
- Databricks services that integrate with other components of the platform, such as notebooks, jobs, and cluster manager
Databricks Runtime versions are released on a regular basis:
- Major versions are represented by an increment to the version number that precedes the decimal point (the jump from 3.5 to 4.0, for example). They are released when there are major changes, some of which may not be backwards-compatible.
- Feature versions are represented by an increment to the version number that follows the decimal point (the jump from 3.4 to 3.5, for example). Each major release includes multiple feature releases. Feature releases are always backwards compatible with previous releases within their major release.
- Long Term Support versions are represented by an LTS qualifier (for example, 3.5 LTS). For each major release, we declare a “canonical” feature version, for which we provide two full years of support. See Databricks runtime support lifecycle for more information.