Databricks Connect release notes

This page lists releases and maintenance updates issued for Databricks Connect.

The Databricks Connect major and minor package version must always match your Databricks Runtime version. Databricks recommends that you always use the most recent patch version of Databricks Connect that matches your Databricks Runtime version. For example, when you use a Databricks Runtime 7.3 cluster, use the latest databricks-connect==7.3.* package.

Databricks Connect for Databricks Runtime 9.1 LTS

Databricks Connect 9.1.1

October 5, 2021

  • Databricks Connect client update to support Databricks Runtime 9.1 maintenance release of October 5, 2021.

Databricks Connect 9.1.0

September 23, 2021

Databricks Connect 9.1 is now GA.

Databricks Connect for Databricks Runtime 8.1

Databricks Connect 8.1.14

September 23, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 23, 2021.

Databricks Connect 8.1.13

September 15, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 15, 2021.

Databricks Connect 8.1.12

September 8, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 8, 2021.

Databricks Connect 8.1.11

August 25, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 25, 2021.

Databricks Connect 8.1.10

August 11, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 11, 2021.

Databricks Connect 8.1.9

July 30, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 30, 2021.

Databricks Connect 8.1.8

July 14, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 14, 2021.

Databricks Connect 8.1.7

June 29, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 29, 2021.

Databricks Connect 8.1.6

June 15, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 15, 2021.

Databricks Connect 8.1.5

June 7, 2021

  • [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

Databricks Connect 8.1.4

May 26, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of May 26, 2021.

Databricks Connect 8.1.3

April 30, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 30, 2021.

Databricks Connect 8.1.2

April 27, 2021

  • Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 27, 2021.

Databricks Connect 8.1.1

April 7, 2021

  • The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
  • Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
  • [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

Databricks Connect 8.1.0

March 22, 2021

Databricks Connect 8.1 is now GA.

  • Improved support of Catalog API V2 commands
  • Improved support of SQL expressions in ANSI mode

Databricks Connect for Databricks Runtime 7.3 LTS

Databricks Connect 7.3.26

October 5, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of October 5, 2021.

Databricks Connect 7.3.25

September 23, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of September 23, 2021.

Databricks Connect 7.3.24

September 15, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of September 15, 2021.

Databricks Connect 7.3.23

September 8, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of September 8, 2021.

Databricks Connect 7.3.22

August 25, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of August 25, 2021.

Databricks Connect 7.3.21

August 11, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of August 11, 2021.

Databricks Connect 7.3.20

July 30, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of July 30, 2021.

Databricks Connect 7.3.19

July 14, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of July 14, 2021.

Databricks Connect 7.3.18

June 29, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of June 29, 2021.

Databricks Connect 7.3.17

June 15, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of June 15, 2021.

Databricks Connect 7.3.16

June 7, 2021

  • [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

Databricks Connect 7.3.15

May 26, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of May 26, 2021.

Databricks Connect 7.3.14

April 30, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of April 30, 2021.

Databricks Connect 7.3.13

April 27, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of April 27, 2021.

Databricks Connect 7.3.12

April 7, 2021

  • The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
  • Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
  • [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

Databricks Connect 7.3.11

March 24, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of March 24, 2021

Databricks Connect 7.3.10

March 9, 2021

  • [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir

  • [ES-68776] Update Azure Blob File System client

  • Add support for FlatMapCoGroupsInPandas

  • Make initial connection process during cluster startup more robust

    Note

    The Databricks Runtime cluster must be running the latest maintenance release, and the Databricks Connect client must be updated to version 7.3.10 to continue to use the Azure Blob File System and mapPartitions and mapElements RDD operations with Databricks Connect. A restart of the Databricks Runtime cluster is required to update it to the latest maintenance release.

Databricks Connect 7.3.9

February 23, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of February 23, 2021

Databricks Connect 7.3.8

February 4, 2021

  • [ES-58153] Fix Analysis Exception when creating tables or views based on a query with aggregations.

Databricks Connect 7.3.7

January 20, 2021

  • Databricks Connect client update to support Databricks Runtime 7.3 maintenance release of January 20, 2021

Databricks Connect 7.3.6

January 12, 2021

  • [ES-59054] Fix Delta Optimize Table command through Databricks Connect
  • [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 7.3.5

December 9, 2020

Databricks Connect 7.3 is now GA.

  • [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

Databricks Connect 7.3.4 Beta

November 24, 2020

  • [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a terminated Databricks Runtime cluster.
  • Fix an error where client could not synchronize state with server correctly if INFO logging level was disabled.

Databricks Connect 7.3.3 Beta

November 3, 2020

Initial Databricks Connect release for Databricks Runtime 7.3. This release includes:

  • Support for Delta Lake time travel.
  • Easy transition between Databricks Connect clients and Databricks Runtime jobs or notebooks when using DBUtils. See Access DBUtils.

Known issues:

Databricks Runtime version validation is disabled in this beta release. Make sure to connect this client to a 7.3 Databricks Runtime cluster.

Databricks Connect for Databricks Runtime 7.1 (Unsupported)

Databricks Connect 7.1.14

February 4, 2021

  • Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of February 4, 2021

Databricks Connect 7.1.13

January 20, 2021

  • Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of January 20, 2021

Databricks Connect 7.1.12

January 12, 2021

  • [ES-59054] Fix Delta Optimize Table command through Databricks Connect
  • [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 7.1.11

December 9, 2020

  • [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

Databricks Connect 7.1.10

November 24, 2020

  • Prevent status polls from restarting the cluster.
  • [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
  • [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
  • [ES-34186] Avoid out of memory exception related to logging.
  • [ES-39493] Prevent recursive load error in databricks-connect test.
  • [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a Databricks Runtime cluster that is terminated.
  • Fix an error in which client could not synchronize state with server correctly if INFO logging level was disabled.
  • Minor fixes.

Databricks Connect 7.1.1

September 23, 2020

  • [ES-32536] Fix an issue with serialization of nested timezone-aware expressions.
  • [ES-33705] Support Azure Data Lake Storage Gen2 filesystem in DButils.

Databricks Connect 7.1.0

August 13, 2020

Initial Databricks Connect release for Databricks Runtime 7.1.

  • Support new Spark 3.0 DataSource V2 APIs.
  • Support using multiple SparkSessions in a single Databricks Connect application.
  • Optimize Databricks Connect state synchronization between client and server. Avoid excessive polling for status updates when the server is idle.

Databricks Connect for Databricks Runtime 6.4

Databricks Connect 6.4.28

April 7, 2021

  • The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.

Databricks Connect 6.4.27

March 24, 2021

  • Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of March 24, 2021

Databricks Connect 6.4.26

March 15, 2021

  • New version has been released, but there are no changes compared to Databricks Connect 6.4.25

Databricks Connect 6.4.25

March 9, 2021

  • [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir
  • Add support for FlatMapCoGroupsInPandas

Databricks Connect 6.4.24

February 23, 2021

  • Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 23, 2021

Databricks Connect 6.4.23

February 4, 2021

  • Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 4, 2021

Databricks Connect 6.4.22

January 12, 2021

  • [ES-59054] Fix Delta Optimize Table command through Databricks Connect
  • [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

Databricks Connect 6.4.21

December 9, 2020

  • [ES-25890] Fix an error where in some cases DataFrame cache() would have no effect and the query would be executed again.
  • [ES-26196] Support using NATURAL and USING syntax in joins.
  • [ES-33475][ES-34186] Avoid out of memory exception related to logging.
  • [ES-39493] Prevent recursive load error in databricks-connect test.
  • [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
  • [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
  • Minor fixes

Databricks Connect 6.4.2

November 5, 2020

  • [ES-47216] Fix version check compatibility issue with Databricks Runtime 6.4 November 3, 2020 maintenance update.

Databricks Connect 6.4.1

April 8, 2020

Minor fixes and updates.

Databricks Connect 6.4.0

March 7, 2020

Initial Databricks Connect release for Databricks Runtime 6.4.