July 2022

These features and Databricks platform improvements were released in July 2022.

Note

Releases are staged. Your Databricks account may not be updated until a week or more after the initial release date.

Databricks Runtime 11.1 and 11.1 ML are GA

July 27, 2022

Databricks Runtime 11.1 and Databricks Runtime 11.1 ML are now generally available.

See Databricks Runtime 11.1 (EoS) and Databricks Runtime 11.1 for Machine Learning (EoS).

Photon is GA

July 27, 2022

Photon is now generally available, beginning with Databricks Runtime 11.1.

Photon is the native vectorized query engine on Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code. Photon is developed in C++ to take advantage of modern hardware, and uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications—all natively on your data lake.

Photon is part of a high-performance runtime that runs your existing SQL and DataFrame API calls faster and reduces your total cost per workload. Photon is used by default in Databricks SQL warehouses.

See Photon is GA in the Databricks Runtime 11.1 release notes.

Notification upon notebook completion

July 27 - August 15, 2022

If you navigate away from a notebook while it’s running, a notification now appears when it finishes. You can enable or disable these notifications in your browser.

Increased limit for the number of jobs in your Databricks workspaces

July 19-22, 2022

You can now create up to 10,000 jobs in a Databricks workspace when you enable the Increased number of jobs toggle in the Databricks admin console. This feature also enables a paginated Jobs list when you view jobs in the Jobs user interface, reducing the time required to load the Jobs list view.

Databricks recommends Jobs API 2.1 if you enable the increased jobs limit. Using the Jobs 2.0 API results in degraded performance when returning large numbers of jobs.

Verbose audit logs now record when Databricks SQL queries are run

July 14, 2022

You can now configure audit logs to record when Databricks SQL queries are run. To do so, use the workspace configuration setting verbose audit logs. See Enable verbose audit logs. There is one audit log record for submitting a query and another record for query completion (including cancellation).

If verbose audit logs were already enabled for your workspace to log notebook commands, your verbose audit logs automatically now include Databricks SQL queries for the workspace.

Databricks SQL Serverless supports instance profiles whose name does not match its associated role

July 14, 2022

Databricks SQL Serverless now supports instance profiles whose role name and the instance profile name do not match. You can now edit your instance profile and add an explicit IAM role ARN. For related information, see Enable serverless SQL warehouses.

Additionally, the Instance Profile API changed. When creating the instance profile, you can now add an explicit IAM role ARN. There is also a new edit request that you can call to edit an instance profile, but only to add or change the affiliated IAM role ARN.

Configure your workspace to use IMDS v2 (Public Preview)

July 14, 2022

If you want to read instance metadata in your notebooks, AWS provides an API called the IMDS API to read this instance metadata. AWS announced IMDS version 2 (IMDSv2), which includes some security improvements and a new session-oriented flow with requests protected by session authentication.

You can now configure your workspace to enforce the use of IMDS v2 with a new workspace admin setting that is available as Public Preview.

Databricks JDBC driver 2.6.27

July 11, 2022

We have released version 2.6.27 of the Databricks JDBC driver (download). This release upgrades the third-party libraries such as Apache Arrow 7.0.0. This release also includes updates to the license file and configuration installation guide.

In addition, this release resolves the following issues:

In some cases, when an HTTP connection has been active for a long period of time, the connector returns a NoHttpResponseException error.
In some cases, unshaded Log4j classes in the connector JAR file conflict with files deployed in the end user’s environment. The unshaded classes have been removed from the connector JAR file.
When using a DataSource class, the subprotocol names are incorrect.

Databricks ODBC driver 2.6.25

July 11, 2022

We have released version 2.6.25 of the Databricks ODBC driver (download). This release adds support to configure the connector to mitigate straggling result file download operations.

Databricks Runtime 11.1 (Beta)

July 8, 2022

Databricks Runtime 11.1, 11.1 Photon, and 11.1 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 11.1 (EoS) and Databricks Runtime 11.1 for Machine Learning (EoS).

Improved notebook visualizations

July 5-11, 2022: Version 3.75

The built-in support for charts and visualizations in notebooks now supports multiple charts and data profiles. When you run a SQL cell in a notebook or use the display function on a DataFrame, the notebook shows tabbed results, similar to the results display in Databricks SQL. You can now create the same rich visualizations in a notebook that you can create in Databricks SQL. For details, see Visualizations in Databricks notebooks.