Amazon
Web Services
Microsoft Azure
Google Cloud PlatformAugust 2018
These features and Databricks platform improvements were released in August 2018.
Note
Releases are staged. Your Databricks account may not be updated until a week after the initial release date.
Workspace sidebar redesign
August 28 - September 4, 2018: Version 2.79
The sidebar for navigating workspace files has been simplified and streamlined, showing only two swimlanes at a time, providing left and right arrows to scroll horizontally when there are more than two swimlanes to view, defaulting to your own user file view when you click the Workspace or Home button in the left navigation, and enabling you to pin the sidebar open while you work in the main pane, which remains unobstructed. We hope you enjoy the changes.
New environment variables in init scripts
August 28 - September 4, 2018: Version 2.79
Cluster-scoped init scripts now support the following environment variables:
DB_DRIVER_IP- the IP address of the driver nodeDB_INSTANCE_TYPE- the type of Spark nodeDB_PYTHON_VERSION- the version of Python used on the clusterDB_IS_JOB_CLUSTER- indicates whether the cluster was created to run a job
EBS leaked volumes logging and deletion
August 28 - September 4, 2018: Version 2.79
Whenever a Databricks cluster instance is terminated, AWS should automatically delete all EBS volumes associated with the cluster. Unfortunately, if an instance dies during startup, the EBS volumes never get attached to the instance, and AWS never cleans them up. The AWS support team has confirmed that they are working on fix for this bug, but they have not provided an ETA. Until AWS provides a fix, Databricks is providing a workaround that enables you or Databricks support to locate and delete the unattached EBS volumes.
For instructions, see the KnowledgeBase article EBS Leaked Volumes.
Important
In late September or October, Databricks will start deleting all leaked EBS volumes by default. We recommend that you identify any leaked EBS volumes by contacting Databricks support or by using the instructions in the KnowledgeBase article, and save any data on these volumes that you want to keep.
AWS r3 and c3 instance types now deprecated
August 28, 2018
Databricks will remove support for AWS r3 and c3 instance types in one year. At that point they will no longer be available for selection when you create a cluster. API workloads will continue to run after support ceases, but we don’t recommend using these instance types for production workloads at that point.
Audit logging for ACL changes
August 14-21, 2018: Version 2.78
All changes to ACLs are now included in the usage and audit logs, including ACL changes for:
Notebooks and folders
Clusters
Jobs
IAM roles (instance profiles)
See Audit log reference.
Cluster-scoped init scripts
August 14-21, 2018: Version 2.78
Cluster-scoped init scripts are init scripts defined in a cluster configuration. Cluster-scoped init scripts have the following advantages over other types of scripts:
Apply to both clusters you create and those created to run jobs.
Are protected by cluster ACLs.
Log start and finish events in the cluster event log.
Have a configurable log location.
Support pre-defined environment variables, which enable conditional processing.
For details, see Cluster-scoped init scripts.
