November 2018

These features and Databricks platform improvements were released in November 2018.

Note

Releases are staged. Your Databricks account may not be updated until up to a week after the initial release date.

Library UI

Important

This update was reverted on December 7, 2018.

November 27-December 4, 2018: Version 2.85

In this release, the library UI has been significantly improved.

The Databricks UI now supports workspace libraries and cluster-attached libraries. A workspace library exists in the Workspace and can be attached to one or more clusters. A cluster-attached library is a library that exists only in the context of the cluster that it is attached to. In addition:

  • You can now create a library from a file uploaded to object storage.

  • You can now attach and detach libraries from the library details page and a cluster’s Libraries tab.

  • Libraries installed using the API now display in a cluster’s Libraries tab.

Custom Spark heap memory settings enabled

November 27-December 4, 2018: Version 2.85

The following Spark memory settings now take effect:

  • spark.executor.memory

  • spark.driver.memory

Important

  • Databricks has services running on each node so the maximum allowable memory for Spark is less than the memory capacity of the VM reported by the cloud provider. If you want to provide Spark with the maximum amount of heap memory for the executor or driver, don’t specify spark.executor.memory or spark.driver.memory respectively.

  • Some cluster configurations that were previously invalid but ignored may result in cluster failures.

Jobs and idle execution context eviction

November 27-December 4, 2018: Version 2.85

Jobs now auto-evict idle execution contexts. To minimize auto-eviction, Databricks recommends that you use different clusters for jobs and interactive workloads.

Databricks Runtime 5.0 for Machine Learning (Beta) release

November 19, 2018

Databricks Runtime 5.0 ML (Beta) provides a ready-to-go environment for machine learning and data science. It contains multiple popular libraries, including TensorFlow, Keras, and XGBoost. It also supports distributed TensorFlow training using Horovod. Databricks Runtime 5.0 ML is built on top of Databricks Runtime 5.0. Databricks Runtime 5.0 ML includes the following new features:

See the complete release notes for Databricks Runtime 5.0 ML (EoS).

Databricks Runtime 5.0 release

November 8, 2018

Databricks Runtime 5.0 is now available. Databricks Runtime 5.0 includes Apache Spark 2.4.0, new Delta Lake and Structured Streaming features and upgrades, and upgraded Python, R, and Java and Scala libraries. For details, see Databricks Runtime 5.0 (EoS).

On Databricks Runtime 5.0, Databricks now evicts idle execution contexts once a cluster has reached the maximum context limit (145).

displayHTML support for unrestricted loading of third-party content

November 6-13, 2018: Version 2.84

Previously the displayHTML iframe sandbox was missing the allow-same-origin attribute. This meant that the iframe had a null origin, which wasn’t friendly to cross-origin XHR requests, cookies, or accessing embedded iframes. With this release, the displayHTML iframe is served from a new domain, databricksusercontent.com, and the iframe sandbox now includes the allow-same-origin attribute.

There is no need to change your usage of displayHTML if it’s already working for you.

databricksusercontent.com will need to be accessible from your browser. If it is currently blocked by your corporate network, it will need to be whitelisted by IT.