August 2018

These features and Databricks platform improvements were released in August 2018.

Note

Releases are staged. Your Databricks account may not be updated until a week after the initial release date.

Workspace sidebar redesign

August 28 - September 4, 2018: Version 2.79

The sidebar for navigating workspace files has been simplified and streamlined, showing only two swimlanes at a time, providing left and right arrows to scroll horizontally when there are more than two swimlanes to view, defaulting to your own user file view when you click the Workspace or Home button in the left navigation, and enabling you to pin the sidebar open while you work in the main pane, which remains unobstructed. We hope you enjoy the changes.

../../../_images/sidebar.png

New environment variables in init scripts

August 28 - September 4, 2018: Version 2.79

Cluster-scoped init scripts now support the following environment variables:

  • DB_DRIVER_IP - the IP address of the driver node
  • DB_INSTANCE_TYPE - the type of Spark node
  • DB_PYTHON_VERSION - the version of Python used on the cluster
  • DB_IS_JOB_CLUSTER - indicates whether the cluster was created to run a job

See Cluster-scoped init scripts.

EBS leaked volumes logging and deletion

August 28 - September 4, 2018: Version 2.79

Whenever a Databricks cluster instance is terminated, AWS should automatically delete all EBS volumes associated with the cluster. Unfortunately, if an instance dies during startup, the EBS volumes never get attached to the instance, and AWS never cleans them up. The AWS support team has confirmed that they are working on fix for this bug, but they have not provided an ETA. Until AWS provides a fix, Databricks is providing a workaround that enables you or Databricks support to locate and delete the unattached EBS volumes.

For instructions, see EBS Leaked Volumes.

Important

In late September or October, Databricks will start deleting all leaked EBS volumes by default. We recommend that you identify any leaked EBS volumes by contacting Databricks support or by using the instructions in EBS Leaked Volumes, and save any data on these volumes that you want to keep.

AWS r3 and c3 instance types now deprecated

August 28, 2018

Databricks will remove support for AWS r3 and c3 instance types in one year. At that point they will no longer be available for selection when you create a cluster. API workloads will continue to run after support ceases, but we don’t recommend using these instance types for production workloads at that point.

Audit logging for ACL changes

August 14-21, 2018: Version 2.78

All changes to ACLs are now included in the usage and audit logs, including ACL changes for:

  • Notebooks and folders
  • Clusters
  • Jobs
  • IAM roles

See Audit Logs.

Cluster-scoped init scripts

August 14-21, 2018: Version 2.78

Cluster-scoped init scripts are init scripts defined in a cluster configuration. Cluster-scoped init scripts have the following advantages over other types of scripts:

  • Apply to both clusters you create and those created to run jobs.
  • Are protected by cluster ACLs.
  • Log start and finish events in the cluster event log.
  • Have a configurable log location.
  • Support pre-defined environment variables, which enable conditional processing.

For details, see Cluster-scoped init scripts.

Collapsible headings

August 14-21, 2018: Version 2.78

Cells that appear after cells containing Markdown headings can now be collapsed into the heading cell. The following image shows a Heading 1 with the following two cells collapsed into it.

../../../_images/headings.png

Click the + and - to expand and collapse headings.