November 2019

These features and Databricks platform improvements were released in November 2019.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 6.2 ML Beta

November 15, 2019

Databricks Runtime 6.2 ML Beta brings many library upgrades, including:

  • TensorFlow and TensorBoard: 1.14.0 to 1.15.0.

  • PyTorch: 1.2.0 to 1.3.0.

  • tensorboardX: 1.8 to 1.9.

  • MLflow: 1.3.0 to 1.4.0.

  • Hyperopt: 0.2-db1 with Databricks MLflow integrations.

  • mleap-databricks-runtime to 0.15.0 and includes mleap-xgboost-runtime.

For more information, see the complete Databricks Runtime 6.2 for ML (EoS) release notes.

Databricks Runtime 6.2 Beta

November 15, 2019

Databricks Runtime 6.2 Beta brings new features, improvements, and many bug fixes, including:

  • Delta Lake insert-only merge optimized

  • Redshift connector adds multi-region support for reads

For more information, see the complete Databricks Runtime 6.2 (EoS) release notes.

Configure clusters with your own container image using Databricks Container Services

November 7 - 19, 2019: Version 3.6

Generally available in Databricks Runtime 6.1 and Databricks Platform version 3.6, Databricks Container Services lets you configure a cluster with your own container image. You can pre-package complex environments within a container, publish it to a popular container registry such as ACR, ECR, or Docker Hub, and then have Databricks pull the image to build a cluster. Some example use cases include:

  • Library customization - you have full control over the system libraries you want installed

  • Golden container environment - your Docker image is a locked down environment that will never change

  • Docker CI/CD integration - you can integrate Databricks with your Docker CI/CD pipelines

There are many other use cases, ranging from specifying configuration to installing machine learning packages.

For details, see Customize containers with Databricks Container Service.

Cluster detail now shows only cluster ID in the HTTP path

November 7 - 19, 2019: Version 3.6

When you connect a BI tool to Databricks using JDBC or ODBC, you should use the cluster ID variant of the HTTP path, because it is unique. The cluster name option no longer appears on the JDBC/ODBC tab on the cluster detail page.

Secrets referenced by Spark configuration properties and environment variables (Public Preview)

November 7, 2019

Available in Databricks Runtime 6.1 and above.

As of the November 7 maintenance update of Databricks Runtime 6.1, the ability to reference a secret in a Spark configuration property or environment variable is in Public Preview. For details, see Use a secret in a Spark configuration property or environment variable.