Databricks Runtime 15.1 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 15.1, powered by Apache Spark 3.5.0.

Databricks released this version in April 2024.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Breaking changes

Support for Databricks Runtime 15.0 and Databricks Runtime 15.0 for Machine Learning ends on May 31, 2024 because an upgrade of the pandas library from 1.5.3 to 2.0.3 in 15.0 caused a breaking change in multiple Apache Spark APIs. For details of the breaking changes, see The pandas Python library is downgraded to 1.5.3. To resolve this incompatibility, Databricks has released Databricks Runtime 15.1 and Databricks Runtime 15.1 for Machine Learning. The functionality supported by these releases is the same as the 15.0 releases, but with the pandas version downgraded to 1.5.3.

Any existing clusters or jobs that use Databricks Runtime 15.0 releases will continue to work but will not receive product or security updates. You can no longer configure new compute that uses Databricks Runtime 15.0 or Databricks Runtime 15.0 ML in the Databricks UI. If you are not using features or APIs that require pandas 2.0+, Databricks recommends switching those workloads to Databricks Runtime 15.1.

If you are using features or APIs available only in pandas 2.0+ and are not using any functions affected by this version, Databricks recommends switching to Databricks Runtime 15.1 and upgrading the default pandas package using cluster or notebook-scoped libraries. For example, to upgrade the pandas library in a Databricks notebook, use the following command:

%pip install pandas==2.0.3

The pandas Python library is downgraded to 1.5.3

In this release, the pandas library is downgraded from version 2.0.3 to 1.5.3. This downgrade was required because incompatibilities in the 2.0.3 version of pandas caused errors in some pandas API on Spark functions. The following lists the functions affected by the 2.0.3 version of pandas:

  • pyspark.date_range: The closed parameter is missing, causing errors when this function is called by existing code.

  • The functionality of GroupBy with MultiIndex columns is changed, causing unexpected behavior.

  • Some datetime attributes of DatetimeIndex (for example day, month, year, and so forth) return int32 data types instead of int64 data types.

The PySpark spark.sql() function now fails for invalid argument types

In Databricks Runtime 15.1 and later, the args parameter for named or positional parameters passed to the spark.sql() function must be a dictionary or list. If any other object type is passed, the following error is raised: PySparkTypeError: [INVALID_TYPE] Argument `args` should not be a <arg_type>.

Storing libraries in DBFS root is deprecated and disabled by default

To improve the security of libraries in a Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default starting with Databricks Runtime 15.1. Databricks recommends uploading all libraries, which includes Python libraries, JAR files, and Spark connectors, to workspace files, Unity Catalog volumes, or using a package repository. See Recommendations for uploading libraries.

To re-enable storing libraries in the DBFS root, set the following Spark configuration parameter: spark.databricks.driver.dbfsLibraryInstallationAllowed true.

The default Python version is upgraded from 3.10 to 3.11

With Databricks Runtime 15.1, the default Python version is 3.11.0. For the list of upgraded Python libraries, see Library upgrades.

JDK 11 is removed

As previously announced, JDK 11 is removed from Databricks Runtime 15.1 and later. Databricks recommends upgrading to JDK 17 when upgrading to Databricks Runtime 15.1 and later. See Databricks SDK for Java.

The Python REPL now terminates gracefully

With Databricks Runtime 15.1, the notebook’s Python REPL process now terminates gracefully at exit. This causes any atexit hooks to be honored. For example, in a workflow with multiple Python notebook tasks, atexit hooks registered in the first task execute before the second Python notebook task runs.

New features and improvements

The * (star) clause is now supported in the WHERE clause

You can now use the star (*) clause in the WHERE clause to reference all columns from the SELECT list. For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*).

The spark.sql.json.enablePartialResults configuration is now enabled by default

The Spark configuration spark.sql.json.enablePartialResults is now enabled by default, allowing the JSON parser to correctly handle parsing errors for complex types such as struct, map, and array without dropping the remaining fields. This change fixes a consistency issue for JSON data sources with column pruning.

Cluster libraries now support requirements.txt files

You can now install cluster libraries using a requirements.txt file stored in a workspace file or Unity Catalog volume. In Single user and Shared access mode clusters, the requirements.txt file can reference other files. In No isolation shared access mode clusters, only PyPI packages are supported. See Cluster libraries.

Databricks CLI now available in the web terminal

You can now use the Databricks CLI from the web terminal in the notebook. The notebook must be attached to a cluster in Single user or No isolation shared access mode. For details, see Use web terminal and Databricks CLI.

Add default Python repositories to cluster configurations

In Databricks Runtime 15.1 and later, you can configure global pip index-url and extra-index-url parameters for cluster and notebook-scoped library installation when configuring a cluster or defining a cluster policy. To do so, set the environment variables DATABRICKS_PIP_INDEX_URL and DATABRICKS_PIP_EXTRA_INDEX_URL.

Credential passthrough and Hive metastore table access controls are deprecated

Credential passthrough and Hive metastore table access controls are legacy data governance models. Upgrade to Unity Catalog to simplify the security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog?.

Support for credential passthrough and Hive metastore table access controls will be removed in an upcoming DBR version.

Library upgrades

  • Upgraded Python libraries:

    • black from 22.6.0 to 23.3.0

    • boto3 from 1.24.28 to 1.34.39

    • botocore from 1.27.96 to 1.34.39

    • certifi from 2022.12.7 to 2023.7.22

    • cryptography from 39.0.1 to 41.0.3

    • databricks-sdk from 0.1.6 to 0.20.0

    • distlib from 0.3.7 to 0.3.8

    • googleapis-common-protos from 1.62.0 to 1.63.0

    • grpcio from 1.48.2 to 1.60.0

    • grpcio-status from 1.48.1 to 1.60.0

    • importlib-metadata from 4.6.4 to 6.0.0

    • ipykernel from 6.25.0 to 6.25.1

    • ipython from 8.14.0 to 8.15.0

    • ipywidgets from 7.7.2 to 8.0.4

    • jupyter_core from 5.2.0 to 5.3.0

    • jupyterlab-widgets from 1.0.0 to 3.0.5

    • matplotlib from 3.7.0 to 3.7.2

    • pip from 22.3.1 to 23.2.1

    • platformdirs from 2.5.2 to 3.10.0

    • protobuf from 4.24.0 to 4.24.1

    • pyarrow from 8.0.0 to 14.0.1

    • Pygments from 2.11.2 to 2.15.1

    • pyodbc from 4.0.32 to 4.0.38

    • requests from 2.28.1 to 2.31.0

    • s3transfer from 0.6.2 to 0.10.0

    • scikit-learn from 1.1.1 to 1.3.0

    • scipy from 1.10.0 to 1.11.1

    • setuptools from 65.6.3 to 68.0.0

    • statsmodels from 0.13.5 to 0.14.0

    • tenacity from 8.1.0 to 8.2.2

    • tornado from 6.1 to 6.3.2

    • typing_extensions from 4.4.0 to 4.7.1

    • urllib3 from 1.26.14 to 1.26.16

    • virtualenv from 20.16.7 to 20.24.2

    • widgetsnbextension from 3.6.1 to 4.0.5

    • zipp from 1.0.0 to 3.11.0

  • Upgraded R libraries:

    • arrow from 12.0.1 to 14.0.0.2

    • askpass from 1.1 to 1.2.0

    • base from 4.3.1 to 4.3.2

    • brew from 1.0-8 to 1.0-10

    • brio from 1.1.3 to 1.1.4

    • bslib from 0.5.0 to 0.6.1

    • cli from 3.6.1 to 3.6.2

    • commonmark from 1.9.0 to 1.9.1

    • compiler from 4.3.1 to 4.3.2

    • config from 0.3.1 to 0.3.2

    • cpp11 from 0.4.4 to 0.4.7

    • credentials from 1.3.2 to 2.0.1

    • curl from 5.0.1 to 5.2.0

    • data.table from 1.14.8 to 1.15.0

    • datasets from 4.3.1 to 4.3.2

    • DBI from 1.1.3 to 1.2.1

    • dbplyr from 2.3.3 to 2.4.0

    • desc from 1.4.2 to 1.4.3

    • digest from 0.6.33 to 0.6.34

    • dplyr from 1.1.2 to 1.1.4

    • e1071 from 1.7-13 to 1.7-14

    • evaluate from 0.21 to 0.23

    • fansi from 1.0.4 to 1.0.6

    • fontawesome from 0.5.1 to 0.5.2

    • fs from 1.6.2 to 1.6.3

    • future from 1.33.0 to 1.33.1

    • future.apply from 1.11.0 to 1.11.1

    • gargle from 1.5.1 to 1.5.2

    • gert from 1.9.2 to 2.0.1

    • ggplot2 from 3.4.2 to 3.4.4

    • glmnet from 4.1-7 to 4.1-8

    • glue from 1.6.2 to 1.7.0

    • graphics from 4.3.1 to 4.3.2

    • grDevices from 4.3.1 to 4.3.2

    • grid from 4.3.1 to 4.3.2

    • gtable from 0.3.3 to 0.3.4

    • hardhat from 1.3.0 to 1.3.1

    • haven from 2.5.3 to 2.5.4

    • htmltools from 0.5.5 to 0.5.7

    • htmlwidgets from 1.6.2 to 1.6.4

    • httpuv from 1.6.11 to 1.6.14

    • httr from 1.4.6 to 1.4.7

    • httr2 from 0.2.3 to 1.0.0

    • jsonlite from 1.8.7 to 1.8.8

    • knitr from 1.43 to 1.45

    • labeling from 0.4.2 to 0.4.3

    • later from 1.3.1 to 1.3.2

    • lava from 1.7.2.1 to 1.7.3

    • lifecycle from 1.0.3 to 1.0.4

    • listenv from 0.9.0 to 0.9.1

    • lubridate from 1.9.2 to 1.9.3

    • markdown from 1.7 to 1.12

    • methods from 4.3.1 to 4.3.2

    • openssl from 2.0.6 to 2.1.1

    • parallel from 4.3.1 to 4.3.2

    • pkgbuild from 1.4.2 to 1.4.3

    • pkgload from 1.3.2.1 to 1.3.4

    • plyr from 1.8.8 to 1.8.9

    • prettyunits from 1.1.1 to 1.2.0

    • pROC from 1.18.4 to 1.18.5

    • processx from 3.8.2 to 3.8.3

    • prodlim from 2023.03.31 to 2023.08.28

    • progress from 1.2.2 to 1.2.3

    • progressr from 0.13.0 to 0.14.0

    • promises from 1.2.0.1 to 1.2.1

    • ps from 1.7.5 to 1.7.6

    • purrr from 1.0.1 to 1.0.2

    • ragg from 1.2.5 to 1.2.7

    • Rcpp from 1.0.11 to 1.0.12

    • RcppEigen from 0.3.3.9.3 to 0.3.3.9.4

    • readr from 2.1.4 to 2.1.5

    • recipes from 1.0.6 to 1.0.9

    • rematch from 1.0.1 to 2.0.0

    • remotes from 2.4.2 to 2.4.2.1

    • reprex from 2.0.2 to 2.1.0

    • rlang from 1.1.1 to 1.1.3

    • rmarkdown from 2.23 to 2.25

    • RODBC from 1.3-20 to 1.3-23

    • roxygen2 from 7.2.3 to 7.3.1

    • rprojroot from 2.0.3 to 2.0.4

    • Rserve from 1.8-11 to 1.8-13

    • RSQLite from 2.3.1 to 2.3.5

    • sass from 0.4.6 to 0.4.8

    • scales from 1.2.1 to 1.3.0

    • shiny from 1.7.4.1 to 1.8.0

    • sparklyr from 1.8.1 to 1.8.4

    • splines from 4.3.1 to 4.3.2

    • stats from 4.3.1 to 4.3.2

    • stats4 from 4.3.1 to 4.3.2

    • stringi from 1.7.12 to 1.8.3

    • stringr from 1.5.0 to 1.5.1

    • systemfonts from 1.0.4 to 1.0.5

    • tcltk from 4.3.1 to 4.3.2

    • testthat from 3.1.10 to 3.2.1

    • textshaping from 0.3.6 to 0.3.7

    • tidyr from 1.3.0 to 1.3.1

    • timechange from 0.2.0 to 0.3.0

    • timeDate from 4022.108 to 4032.109

    • tinytex from 0.45 to 0.49

    • tools from 4.3.1 to 4.3.2

    • utf8 from 1.2.3 to 1.2.4

    • utils from 4.3.1 to 4.3.2

    • uuid from 1.1-0 to 1.2-0

    • vctrs from 0.6.3 to 0.6.5

    • vroom from 1.6.3 to 1.6.5

    • waldo from 0.5.1 to 0.5.2

    • withr from 2.5.0 to 3.0.0

    • xfun from 0.39 to 0.41

    • xml2 from 1.3.5 to 1.3.6

    • yaml from 2.3.7 to 2.3.8

    • zip from 2.3.0 to 2.3.1

  • Upgraded Java libraries:

    • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.15.1 to 2.16.0

    • com.google.flatbuffers.flatbuffers-java from 1.12.0 to 23.5.26

    • com.typesafe.config from 1.2.1 to 1.4.3

    • org.apache.ant.ant from 1.9.16 to 1.10.11

    • org.apache.ant.ant-jsch from 1.9.16 to 1.10.11

    • org.apache.ant.ant-launcher from 1.9.16 to 1.10.11

    • org.apache.arrow.arrow-format from 12.0.1 to 15.0.0

    • org.apache.arrow.arrow-memory-core from 12.0.1 to 15.0.0

    • org.apache.arrow.arrow-memory-netty from 12.0.1 to 15.0.0

    • org.apache.arrow.arrow-vector from 12.0.1 to 15.0.0

    • org.apache.avro.avro from 1.11.2 to 1.11.3

    • org.apache.avro.avro-ipc from 1.11.2 to 1.11.3

    • org.apache.avro.avro-mapred from 1.11.2 to 1.11.3

    • org.apache.logging.log4j.log4j-1.2-api from 2.20.0 to 2.22.1

    • org.apache.logging.log4j.log4j-api from 2.20.0 to 2.22.1

    • org.apache.logging.log4j.log4j-core from 2.20.0 to 2.22.1

    • org.apache.logging.log4j.log4j-slf4j2-impl from 2.20.0 to 2.22.1

    • org.postgresql.postgresql from 42.6.0 to 42.6.1

Apache Spark

Databricks Runtime 15.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.3 LTS, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-45527] [DBRRM-805][ES-1073714] Revert “[SC-151626][CORE] Use fraction to do the resource calculation”

  • [SPARK-47102] [DBRRM-803][SC-158253][SQL] Add the COLLATION_ENABLED config flag

  • [SPARK-46973] [SC-158140][DBRRM-777][SQL] Skip V2 table lookup when a table is in V1 table cache

  • [SPARK-46808] [SC-154113][PYTHON] Refine error classes in Python with automatic sorting function

  • [SPARK-47251] [SC-158291][PYTHON] Block invalid types from the args argument for sql command

  • [SPARK-47251] [SC-158121][PYTHON] Block invalid types from the args argument for sql command

  • [SPARK-47108] [SC-157204][CORE] Set derby.connection.requireAuthentication to false explicitly in CLIs

  • [SPARK-45599] [SC-157537][CORE] Use object equality in OpenHashSet

  • [SPARK-47099] [SC-157352][SQL] Use ordinalNumber to uniformly set the value of paramIndex for the error class UNEXPECTED_INPUT_TYPE

  • [SPARK-46558] [SC-151363][CONNECT] Extract a helper function to eliminate the duplicate code that retrieves MessageParameters from ErrorParams in GrpcExceptionConverter

  • [SPARK-43117] [SC-156177][CONNECT] Make ProtoUtils.abbreviate support repeated fields

  • [SPARK-46342] [SC-150283][SQL] Replace IllegalStateException by SparkException.internalError in sql

  • [SPARK-47123] [SC-157412][CORE] JDBCRDD does not correctly handle errors in getQueryOutputSchema

  • [SPARK-47189] [SC-157667][SQL] Tweak column error names and text

  • [SPARK-45789] [SC-157101][SQL] Support DESCRIBE TABLE for clustering columns

  • [SPARK-43256] [SC-157699][SQL] Remove error class LEGACYERROR_TEMP_2021

  • [SPARK-47201] [SC-157782][PYTHON][CONNECT] sameSemantics checks input types

  • [SPARK-47183] [SC-157660][PYTHON] Fix the error class for sameSemantics

  • [SPARK-47179] [SC-157663][SQL] Improve error message from SparkThrowableSuite for better debuggability

  • [SPARK-46965] [SC-155791][CORE] Check logType in Utils.getLog

  • [SPARK-46832] [SC-156774][SQL] Introducing Collate and Collation expressions

  • [SPARK-46946] [SC-155721][SQL] Supporting broadcast of multiple filtering keys in DynamicPruning

  • [SPARK-47079] [VARIANT-22][SC-157598][PYTHON][SQL][CONNECT] Add Variant type info to PySpark

  • [SPARK-47101] [SC-157360][SQL] Allow comma to be used in top-level column names and remove check nested type definition in HiveExternalCatalog.verifyDataSchema

  • [SPARK-47173] [SC-157571][SS][UI] Fix a typo in streaming UI explanation

  • [SPARK-47113] [SC-157348][CORE] Revert S3A endpoint fixup logic of SPARK-35878

  • [SPARK-47130] [SC-157462][CORE] Use listStatus to bypass block location info when cleaning driver logs

  • [SPARK-43259] [SC-157368][SQL] Assign a name to the error class LEGACYERROR_TEMP_2024

  • [SPARK-47104] [SC-157355][SQL] TakeOrderedAndProjectExec should initialize the unsafe projection

  • [SPARK-47137] [SC-157463][PYTHON][CONNECT] Add getAll to spark.conf for feature parity with Scala

  • [SPARK-46924] [SC-154905][CORE] Fix Load New button in Master/HistoryServer Log UI

  • [SPARK-47069] [SC-157370][PYTHON][CONNECT] Introduce spark.profile.show/dump for SparkSession-based profiling

  • [SPARK-46812] [SC-157075][SQL][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile

  • [SPARK-46833] [SC-155866][SQL] Collations - Introducing CollationFactory which provides comparison and hashing rules for supported collations

  • [SPARK-47057] [SC-157036][PYTHON] Reeanble MyPy data test

  • [SPARK-46934] [SC-157084][SQL] Read/write roundtrip for struct type with special characters with HMS

  • [SPARK-46727] [SC-153472][SQL] Port classifyException() in JDBC dialects on error classes

  • [SPARK-46964] [SC-155740][SQL] Change the signature of the hllInvalidLgK query execution error to take an integer as 4th argument

  • [SPARK-46949] [SC-155719][SQL] Support CHAR/VARCHAR through ResolveDefaultColumns

  • [SPARK-46972] [SC-155888][SQL] Fix asymmetrical replacement for char/varchar in V2SessionCatalog.createTable

  • [SPARK-47014] [SC-156778][PYTHON][CONNECT] Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession

  • [SPARK-46689] [SC-156221][SPARK-46690][PYTHON][CONNECT] Support v2 profiling in group/cogroup applyInPandas/applyInArrow

  • [SPARK-47040] [SC-156773][CONNECT] Allow Spark Connect Server Script to wait

  • [SPARK-46688] [SC-156113][SPARK-46691][PYTHON][CONNECT] Support v2 profiling in aggregate Pandas UDFs

  • [SPARK-47004] [SC-156499][CONNECT] Added more tests to ClientStreamingQuerySuite to increase Scala client test coverage

  • [SPARK-46994] [SC-156169][PYTHON] Refactor PythonWrite to prepare for supporting python data source streaming write

  • [SPARK-46979] [SC-156587][SS] Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider

  • [SPARK-46960] [SC-155940][SS] Testing Multiple Input Streams with TransformWithState operator

  • [SPARK-46966] [SC-156112][Python] Add UDTF API for ‘analyze’ method to indicate subset of input table columns to select

  • [SPARK-46895] [SC-155950][CORE] Replace Timer with single thread scheduled executor

  • [SPARK-46967] [SC-155815][CORE][UI] Hide Thread Dump and Heap Histogram of Dead executors in Executors UI

  • [SPARK-46512] [SC-155826][CORE] Optimize shuffle reading when both sort and combine are used.

  • [SPARK-46958] [SC-155942][SQL] Add missing timezone to coerce default values

  • [SPARK-46754] [SC-153830][SQL][AVRO] Fix compression code resolution in avro table definition and write options

  • [SPARK-46728] [SC-154207][PYTHON] Check Pandas installation properly

  • [SPARK-43403] [SC-154216][UI] Ensure old SparkUI in HistoryServer has been detached before loading new one

  • [SPARK-39910] [SC-156203][SQL] Delegate path qualification to filesystem during DataSource file path globbing

  • [SPARK-47000] [SC-156123] Revert “[SC-156123][CORE] Use getTotalMemorySize…

  • [SPARK-46819] [SC-154560][CORE] Move error categories and states into JSON

  • [SPARK-46997] [SC-156175][CORE] Enable spark.worker.cleanup.enabled by default

  • [SPARK-46865] [SC-156168][SS] Add Batch Support for TransformWithState Operator

  • [SPARK-46987] [SC-156040][CONNECT] ProtoUtils.abbreviate avoid unnecessary setField operation

  • [SPARK-46970] [SC-155816][CORE] Rewrite OpenHashSet#hasher with pattern matching

  • [SPARK-46984] [SC-156024][PYTHON] Remove pyspark.copy_func

  • [SPARK-47000] [SC-156123][CORE] Use getTotalMemorySize in WorkerArguments

  • [SPARK-46980] [SC-155914][SQL][MINOR] Avoid using internal APIs in dataframe end-to-end tests

  • [SPARK-46931] Revert “[SC-155661][PS] Implement {Frame, Series}.to_hdf”

  • [SPARK-46618] [SC-153828][SQL] Improve error messages for DATA_SOURCE_NOT_FOUND error

  • [SPARK-46849] [SC-154586][SQL] Run optimizer on CREATE TABLE column defaults

  • [SPARK-46911] [SC-155724][SS] Adding deleteIfExists operator to StatefulProcessorHandleImpl

  • [SPARK-43273] [SQL] Support lz4raw compression codec for Parquet

  • [SPARK-47002] [SC-156223][Python] Return better error message if UDTF ‘analyze’ method ‘orderBy’ field accidentally returns a list of strings

  • [SPARK-46890] [SC-155802][SQL] Fix CSV parsing bug with existence default values and column pruning

  • [SPARK-46977] [SC-155892][CORE] A failed request to obtain a token from one NameNode should not skip subsequent token requests

  • [SPARK-46608] [SC-151806][SQL] Restore backward compatibility of JdbcDialect.classifyException

  • [SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause

  • [SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause

  • [SPARK-46852] [SC-155665][SS] Remove use of explicit key encoder and pass it implicitly to the operator for transformWithState operator

  • [SPARK-46864] [SC-155669][SS] Onboard Arbitrary StateV2 onto New Error Class Framework

  • [SPARK-45807] [SC-155706][SQL] Return View after calling replaceView(..)

  • [SPARK-46899] [SC-154651][CORE] Remove POST APIs from MasterWebUI when spark.ui.killEnabled is false

  • [SPARK-46868] [SC-154539][CORE] Support Spark Worker Log UI

  • [SPARK-46931] [SC-155661][PS] Implement {Frame, Series}.to_hdf

  • [SPARK-46940] [SC-155667][CORE] Remove unused updateSparkConfigFromProperties and isAbsoluteURI in o.a.s.u.Utils

  • [SPARK-46929] [SC-155659][CORE][CONNECT][SS] Use ThreadUtils.shutdown to close thread pools

  • [SPARK-46400] [SC-155658][CORE][SQL] When there are corrupted files in the local maven repo, skip this cache and try again

  • [SPARK-46932] [SC-155655] Clean up the imports in pyspark.pandas.test_*

  • [SPARK-46683] [SC-154120] Write a subquery generator that generates subqueries permutations to increase testing coverage

  • [SPARK-46914] [SC-154776][UI] Shorten app name in the summary table on the History Page

  • [SPARK-46831] [SC-154699][SQL] Collations - Extending StringType and PhysicalStringType with collationId field

  • [SPARK-46816] [SC-154415][SS] Add base support for new arbitrary state management operator, single valueState type, multiple state variables and underlying support for column families for RocksDBStateStoreProvider with/without changelog checkpointing

  • [SPARK-46925] [SC-154890][PYTHON][CONNECT] Add a warning that instructs to install memory_profiler for memory profiling

  • [SPARK-46927] [SC-154904][PYTHON] Make assertDataFrameEqual work properly without PyArrow

  • [SPARK-46021] [SC-148987][CORE] Support cancel future jobs belonging to a job group

  • [SPARK-46747] [SC-154789][SQL] Avoid scan in getTableExistsQuery for JDBC Dialects

  • [SPARK-46904] [SC-154704][UI] Fix display issue of History UI summary

  • [SPARK-46870] [SC-154541][CORE] Support Spark Master Log UI

  • [SPARK-46893] [SC-154780][UI] Remove inline scripts from UI descriptions

  • [SPARK-46910] [SC-154760][PYTHON] Eliminate JDK Requirement in PySpark Installation

  • [SPARK-46907] [SC-154736][CORE] Show driver log location in Spark History Server

  • [SPARK-46902] [SC-154661][UI] Fix Spark History Server UI for using un-exported setAppLimit

  • [SPARK-46687] [SC-154725][PYTHON][CONNECT] Basic support of SparkSession-based memory profiler

  • [SPARK-46623] [SC-153831][CORE][MLLIB][SQL] Replace SimpleDateFormat with DateTimeFormatter

  • [SPARK-46875] [SC-154627][SQL] When the mode is null, a NullPointException should not be thrown

  • [SPARK-46872] [SC-154543][CORE] Recover log-view.js to be non-module

  • [SPARK-46883] [SC-154629][CORE] Support /json/clusterutilization API

  • [SPARK-46886] [SC-154630][CORE] Enable spark.ui.prometheus.enabled by default

  • [SPARK-46873] [SC-154638][SS] Do not recreate new StreamingQueryManager for the same Spark Session

  • [SPARK-46829] [SC-154419][CORE] Remove createExecutorEnv from SparkEnv

  • [SPARK-46827] [SC-154280][CORE] Make RocksDBPersistenceEngine to support a symbolic link

  • [SPARK-46903] [SC-154662][CORE] Support Spark History Server Log UI

  • [SPARK-46874] [SC-154646][PYTHON] Remove pyspark.pandas dependency from assertDataFrameEqual

  • [SPARK-46889] [SC-154639][CORE] Validate spark.master.ui.decommission.allow.mode setting

  • [SPARK-46850] [SC-154443][SQL] Convert _LEGACY_ERROR_TEMP_2102 to UNSUPPORTED_DATATYPE

  • [SPARK-46704] [SC-153323][CORE][UI] Fix MasterPage to sort Running Drivers table by Duration column correctly

  • [SPARK-46796] [SC-154279][SS] Ensure the correct remote files (mentioned in metadata.zip) are used on RocksDB version load

  • [SPARK-46888] [SC-154636][CORE] Fix Master to reject /workers/kill/ requests if decommission is disabled

  • [SPARK-46818] [SC-154422][SQL] Improve error messages for Range with non-foldable inputs

  • [SPARK-46898] [SC-154649][CONNECT] Simplify the protobuf function transformation in Planner

  • [SPARK-46828] [SC-154413][SQL] Remove the invalid assertion of remote mode for spark sql shell

  • [SPARK-46733] [SC-154274][CORE] Simplify the BlockManager by the exit operation only depend on interrupt thread.

  • [SPARK-46777] [SC-154016][SS] Refactor StreamingDataSourceV2Relation catalyst structure to be more on-par with the batch version

  • [SPARK-46515] [SC-151716] Add MONTHNAME function

  • [SPARK-46823] [SC-154276][CONNECT][PYTHON] LocalDataToArrowConversion should check the nullability

  • [SPARK-46787] [SC-154404][CONNECT] bloomFilter function should throw AnalysisException for invalid input

  • [SPARK-46779] [SC-154045][SQL] InMemoryRelation instances of the same cached plan should be semantically equivalent

  • [SPARK-45827] [SC-153973] Disallow partitioning on Variant column

  • [SPARK-46797] [SC-154085][CORE] Rename spark.deploy.spreadOut to spark.deploy.spreadOutApps

  • [SPARK-46094] [SC-153421] Support Executor JVM Profiling

  • [SPARK-46746] [SC-153557][SQL][AVRO] Attach codec extension to avro datasource files

  • [SPARK-46698] [SC-153288][CORE] Replace Timer with single thread scheduled executor for ConsoleProgressBar.

  • [SPARK-46629] [SC-153835] Fix for STRUCT type DDL not picking up nullability and comment

  • [SPARK-46750] [SC-153772][CONNECT][PYTHON] DataFrame APIs code clean up

  • [SPARK-46769] [SC-153979][SQL] Refine timestamp related schema inference

  • [SPARK-46765] [SC-153904][PYTHON][CONNECT] Make shuffle specify the datatype of seed

  • [SPARK-46579] [SC-151481][SQL] Redact JDBC url in errors and logs

  • [SPARK-46686] [SC-153875][PYTHON][CONNECT] Basic support of SparkSession based Python UDF profiler

  • [SPARK-46748] Revert “[SC-153800][CORE] Remove *slav**.sh scri…

  • [SPARK-46707] [SC-153833][SQL] Added throwable field to expressions to improve predicate pushdown

  • [SPARK-46519] [SC-151278][SQL] Clear unused error classes from error-classes.json file

  • [SPARK-46677] [SC-153426][SQL][CONNECT] Fix dataframe["*"] resolution

  • [SPARK-46317] [SC-150184][PYTHON][CONNECT] Match minor behaviour matching in SparkSession with full test coverage

  • [SPARK-46748] [SC-153800][CORE] Remove *slav**.sh scripts

  • [SPARK-46663] [SC-153490][PYTHON] Disable memory profiler for pandas UDFs with iterators

  • [SPARK-46410] [SC-150776][SQL] Assign error classes/subclasses to JdbcUtils.classifyException

  • [SPARK-46277] [SC-150126][PYTHON] Validate startup urls with the config being set

  • [SPARK-46612] [SC-153536][SQL] Do not convert array type string retrieved from jdbc driver

  • [SPARK-46254] [SC-149788][PYTHON] Remove stale Python 3.8/3.7 version checking

  • [SPARK-46490] [SC-151335][SQL] Require error classes in SparkThrowable sub-classes

  • [SPARK-46383] [SC-153274][SC-147443][WARMFIX] Reduce Driver Heap Usage by Reducing the Lifespan of TaskInfo.accumulables()

  • [SPARK-46541] [SC-153109][SQL][CONNECT] Fix the ambiguous column reference in self join

  • [SPARK-46381] [SC-150495][SQL] Migrate sub-classes of AnalysisException to error classes

  • [SPARK-46351] [SC-150335][SQL] Require an error class in AnalysisException

  • [SPARK-46220] [SC-149689][SQL] Restrict charsets in decode()

  • [SPARK-46369] [SC-150340][CORE] Remove kill link from RELAUNCHING drivers in MasterPage

  • [SPARK-46052] [SC-153284][CORE] Remove function TaskScheduler.killAllTaskAttempts

  • [SPARK-46536] [SC-153164][SQL] Support GROUP BY calendar_interval_type

  • [SPARK-46675] [SC-153209][SQL] Remove unused inferTimestampNTZ in ParquetReadSupport

  • [SPARK-46717] [SC-153423][CORE] Simplify ReloadingX509TrustManager by the exit operation only depend on interrupt thread.

  • [SPARK-46722] [SC-153438][CONNECT] Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)

  • [SPARK-46187] [SC-149580][SQL] Align codegen and non-codegen implementation of StringDecode

  • [SPARK-46258] [SC-149799][CORE] Add RocksDBPersistenceEngine

  • [SPARK-46216] [SC-149676][CORE] Improve FileSystemPersistenceEngine to support compressions

  • [SPARK-46189] [SC-149567][PS][SQL] Perform comparisons and arithmetic between same types in various Pandas aggregate functions to avoid interpreted mode errors

  • [SPARK-46184] [SC-149563][CORE][SQL][CONNECT][MLLIB] Reduce stack depth by replace Option.isDefined with Option.isEmpty

  • [SPARK-46700] [SC-153329][CORE] Count the last spilling for the shuffle disk spilling bytes metric

  • [SPARK-45642] [SC-150286][CORE][SQL] Fix FileSystem.isFile & FileSystem.isDirectory is deprecated

  • [SPARK-46640] [SC-153272][SQL] Fix RemoveRedundantAlias by excluding subquery attributes

  • [SPARK-46681] [SC-153287][CORE] Refactor ExecutorFailureTracker#maxNumExecutorFailures to avoid calculating defaultMaxNumExecutorFailures when MAX_EXECUTOR_FAILURES is configured

  • [SPARK-46695] [SC-153289][SQL][HIVE] Always setting hive.execution.engine to mr

  • [SPARK-46325] [SC-150228][CONNECT] Remove unnecessary override functions when constructing WrappedCloseableIterator in ResponseValidator#wrapIterator

  • [SPARK-46232] [SC-149699][PYTHON] Migrate all remaining ValueError into PySpark error framework.

  • [SPARK-46547] [SC-153174][SS] Swallow non-fatal exception in maintenance task to avoid deadlock between maintenance thread and streaming aggregation operator

  • [SPARK-46169] [SC-149373][PS] Assign appropriate JIRA numbers for missing parameters from DataFrame API.

  • [SPARK-45857] [SC-148096][SQL] Enforce the error classes in sub-classes of AnalysisException

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 22.04.4 LTS

  • Java: Zulu 8.74.0.17-CA-linux64

  • Scala: 2.12.15

  • Python: 3.11.0

  • R: 4.3.2

  • Delta Lake: 3.1.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

asttokens

2.0.5

astunparse

1.6.3

azure-core

1.30.1

azure-storage-blob

12.19.1

azure-storage-file-datalake

12.14.0

backcall

0.2.0

black

23.3.0

blinker

1.4

boto3

1.34.39

botocore

1.34.39

cachetools

5.3.3

certifi

2023.7.22

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

cloudpickle

2.2.1

comm

0.1.2

contourpy

1.0.5

cryptography

41.0.3

cycler

0.11.0

Cython

0.29.32

databricks-sdk

0.20.0

dbus-python

1.2.18

debugpy

1.6.7

decorator

5.1.1

distlib

0.3.8

entrypoints

0.4

executing

0.8.3

facets-overview

1.1.1

filelock

3.13.1

fonttools

4.25.0

gitdb

4.0.11

GitPython

3.1.42

google-api-core

2.17.1

google-auth

2.28.2

google-cloud-core

2.4.1

google-cloud-storage

2.15.0

google-crc32c

1.5.0

google-resumable-media

2.7.0

googleapis-common-protos

1.63.0

grpcio

1.60.0

grpcio-status

1.60.0

httplib2

0.20.2

idna

3.4

importlib-metadata

6.0.0

ipyflow-core

0.0.198

ipykernel

6.25.1

ipython

8.15.0

ipywidgets

8.0.4

isodate

0.6.1

jedi

0.18.1

jeepney

0.7.1

jmespath

0.10.0

joblib

1.2.0

jupyter_client

7.4.9

jupyter_core

5.3.0

jupyterlab-widgets

3.0.5

keyring

23.5.0

kiwisolver

1.4.4

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

matplotlib

3.7.2

matplotlib-inline

0.1.6

mlflow-skinny

2.11.1

more-itertools

8.10.0

mypy-extensions

0.4.3

nest-asyncio

1.5.6

numpy

1.23.5

oauthlib

3.2.0

packaging

23.2

pandas

1.5.3

parso

0.8.3

pathspec

0.10.3

patsy

0.5.3

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.4.0

pip

23.2.1

platformdirs

3.10.0

plotly

5.9.0

prompt-toolkit

3.0.36

protobuf

4.24.1

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

14.0.1

pyasn1

0.4.8

pyasn1-modules

0.2.8

pyccolo

0.0.52

pycparser

2.21

pydantic

1.10.6

Pygments

2.15.1

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.38

pyparsing

3.0.9

python-dateutil

2.8.2

python-lsp-jsonrpc

1.1.1

pytz

2022.7

PyYAML

6.0

pyzmq

23.2.0

requests

2.31.0

rsa

4.9

s3transfer

0.10.0

scikit-learn

1.3.0

scipy

1.11.1

seaborn

0.12.2

SecretStorage

3.3.1

setuptools

68.0.0

six

1.16.0

smmap

5.0.1

sqlparse

0.4.4

ssh-import-id

5.11

stack-data

0.2.0

statsmodels

0.14.0

tenacity

8.2.2

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tornado

6.3.2

traitlets

5.7.1

typing_extensions

4.7.1

tzdata

2022.1

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.16

virtualenv

20.24.2

wadllib

1.3.6

wcwidth

0.2.5

wheel

0.38.4

widgetsnbextension

4.0.5

zipp

3.11.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.

Library

Version

Library

Version

Library

Version

arrow

14.0.0.2

askpass

1.2.0

assertthat

0.2.1

backports

1.4.1

base

4.3.2

base64enc

0.1-3

bigD

0.2.0

bit

4.0.5

bit64

4.0.5

bitops

1.0-7

blob

1.2.4

boot

1.3-28

brew

1.0-10

brio

1.1.4

broom

1.0.5

bslib

0.6.1

cachem

1.0.8

callr

3.7.3

caret

6.0-94

cellranger

1.1.0

chron

2.3-61

class

7.3-22

cli

3.6.2

clipr

0.8.0

clock

0.7.0

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.9.1

compiler

4.3.2

config

0.3.2

conflicted

1.2.0

cpp11

0.4.7

crayon

1.5.2

credentials

2.0.1

curl

5.2.0

data.table

1.15.0

datasets

4.3.2

DBI

1.2.1

dbplyr

2.4.0

desc

1.4.3

devtools

2.4.5

diagram

1.6.5

diffobj

0.3.5

digest

0.6.34

downlit

0.4.3

dplyr

1.1.4

dtplyr

1.3.1

e1071

1.7-14

ellipsis

0.3.2

evaluate

0.23

fansi

1.0.6

farver

2.1.1

fastmap

1.1.1

fontawesome

0.5.2

forcats

1.0.0

foreach

1.5.2

foreign

0.8-85

forge

0.2.0

fs

1.6.3

future

1.33.1

future.apply

1.11.1

gargle

1.5.2

generics

0.1.3

gert

2.0.1

ggplot2

3.4.4

gh

1.4.0

git2r

0.33.0

gitcreds

0.1.2

glmnet

4.1-8

globals

0.16.2

glue

1.7.0

googledrive

2.1.1

googlesheets4

1.1.1

gower

1.0.1

graphics

4.3.2

grDevices

4.3.2

grid

4.3.2

gridExtra

2.3

gsubfn

0.7

gt

0.10.1

gtable

0.3.4

hardhat

1.3.1

haven

2.5.4

highr

0.10

hms

1.1.3

htmltools

0.5.7

htmlwidgets

1.6.4

httpuv

1.6.14

httr

1.4.7

httr2

1.0.0

ids

1.0.1

ini

0.3.1

ipred

0.9-14

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.8

juicyjuice

0.1.0

KernSmooth

2.23-21

knitr

1.45

labeling

0.4.3

later

1.3.2

lattice

0.21-8

lava

1.7.3

lifecycle

1.0.4

listenv

0.9.1

lubridate

1.9.3

magrittr

2.0.3

markdown

1.12

MASS

7.3-60

Matrix

1.5-4.1

memoise

2.0.1

methods

4.3.2

mgcv

1.8-42

mime

0.12

miniUI

0.1.1.1

mlflow

2.10.0

ModelMetrics

1.2.2.2

modelr

0.1.11

munsell

0.5.0

nlme

3.1-163

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.1.1

parallel

4.3.2

parallelly

1.36.0

pillar

1.9.0

pkgbuild

1.4.3

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.4

plogr

0.2.0

plyr

1.8.9

praise

1.0.0

prettyunits

1.2.0

pROC

1.18.5

processx

3.8.3

prodlim

2023.08.28

profvis

0.3.8

progress

1.2.3

progressr

0.14.0

promises

1.2.1

proto

1.0.0

proxy

0.4-27

ps

1.7.6

purrr

1.0.2

R6

2.5.1

ragg

1.2.7

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.12

RcppEigen

0.3.3.9.4

reactable

0.4.4

reactR

0.5.0

readr

2.1.5

readxl

1.4.3

recipes

1.0.9

rematch

2.0.0

rematch2

2.1.2

remotes

2.4.2.1

reprex

2.1.0

reshape2

1.4.4

rlang

1.1.3

rmarkdown

2.25

RODBC

1.3-23

roxygen2

7.3.1

rpart

4.1.21

rprojroot

2.0.4

Rserve

1.8-13

RSQLite

2.3.5

rstudioapi

0.15.0

rversions

2.1.2

rvest

1.0.3

sass

0.4.8

scales

1.3.0

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.8.0

sourcetools

0.1.7-1

sparklyr

1.8.4

spatial

7.3-15

splines

4.3.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.3.2

stats4

4.3.2

stringi

1.8.3

stringr

1.5.1

survival

3.5-5

swagger

3.33.1

sys

3.4.2

systemfonts

1.0.5

tcltk

4.3.2

testthat

3.2.1

textshaping

0.3.7

tibble

3.2.1

tidyr

1.3.1

tidyselect

1.2.0

tidyverse

2.0.0

timechange

0.3.0

timeDate

4032.109

tinytex

0.49

tools

4.3.2

tzdb

0.4.0

urlchecker

1.0.1

usethis

2.2.2

utf8

1.2.4

utils

4.3.2

uuid

1.2-0

V8

4.4.1

vctrs

0.6.5

viridisLite

0.4.2

vroom

1.6.5

waldo

0.5.2

whisker

0.4.1

withr

3.0.0

xfun

0.41

xml2

1.3.6

xopen

1.0.0

xtable

1.8-4

yaml

2.3.8

zeallot

0.1.0

zip

2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.390

com.amazonaws

aws-java-sdk-cloudformation

1.12.390

com.amazonaws

aws-java-sdk-cloudfront

1.12.390

com.amazonaws

aws-java-sdk-cloudhsm

1.12.390

com.amazonaws

aws-java-sdk-cloudsearch

1.12.390

com.amazonaws

aws-java-sdk-cloudtrail

1.12.390

com.amazonaws

aws-java-sdk-cloudwatch

1.12.390

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.390

com.amazonaws

aws-java-sdk-codedeploy

1.12.390

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.390

com.amazonaws

aws-java-sdk-cognitosync

1.12.390

com.amazonaws

aws-java-sdk-config

1.12.390

com.amazonaws

aws-java-sdk-core

1.12.390

com.amazonaws

aws-java-sdk-datapipeline

1.12.390

com.amazonaws

aws-java-sdk-directconnect

1.12.390

com.amazonaws

aws-java-sdk-directory

1.12.390

com.amazonaws

aws-java-sdk-dynamodb

1.12.390

com.amazonaws

aws-java-sdk-ec2

1.12.390

com.amazonaws

aws-java-sdk-ecs

1.12.390

com.amazonaws

aws-java-sdk-efs

1.12.390

com.amazonaws

aws-java-sdk-elasticache

1.12.390

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.390

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.390

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.390

com.amazonaws

aws-java-sdk-emr

1.12.390

com.amazonaws

aws-java-sdk-glacier

1.12.390

com.amazonaws

aws-java-sdk-glue

1.12.390

com.amazonaws

aws-java-sdk-iam

1.12.390

com.amazonaws

aws-java-sdk-importexport

1.12.390

com.amazonaws

aws-java-sdk-kinesis

1.12.390

com.amazonaws

aws-java-sdk-kms

1.12.390

com.amazonaws

aws-java-sdk-lambda

1.12.390

com.amazonaws

aws-java-sdk-logs

1.12.390

com.amazonaws

aws-java-sdk-machinelearning

1.12.390

com.amazonaws

aws-java-sdk-opsworks

1.12.390

com.amazonaws

aws-java-sdk-rds

1.12.390

com.amazonaws

aws-java-sdk-redshift

1.12.390

com.amazonaws

aws-java-sdk-route53

1.12.390

com.amazonaws

aws-java-sdk-s3

1.12.390

com.amazonaws

aws-java-sdk-ses

1.12.390

com.amazonaws

aws-java-sdk-simpledb

1.12.390

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.390

com.amazonaws

aws-java-sdk-sns

1.12.390

com.amazonaws

aws-java-sdk-sqs

1.12.390

com.amazonaws

aws-java-sdk-ssm

1.12.390

com.amazonaws

aws-java-sdk-storagegateway

1.12.390

com.amazonaws

aws-java-sdk-sts

1.12.390

com.amazonaws

aws-java-sdk-support

1.12.390

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.390

com.amazonaws

jmespath-java

1.12.390

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.17.1

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.15.2

com.fasterxml.jackson.core

jackson-core

2.15.2

com.fasterxml.jackson.core

jackson-databind

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-yaml

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.16.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.15.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.15.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.5-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.10.1

com.google.crypto.tink

tink

1.9.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

23.5.26

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.4.3

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.16.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.13.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.25

io.delta

delta-sharing-client_2.12

1.0.4

io.dropwizard.metrics

metrics-annotation

4.2.19

io.dropwizard.metrics

metrics-core

4.2.19

io.dropwizard.metrics

metrics-graphite

4.2.19

io.dropwizard.metrics

metrics-healthchecks

4.2.19

io.dropwizard.metrics

metrics-jetty9

4.2.19

io.dropwizard.metrics

metrics-jmx

4.2.19

io.dropwizard.metrics

metrics-json

4.2.19

io.dropwizard.metrics

metrics-jvm

4.2.19

io.dropwizard.metrics

metrics-servlets

4.2.19

io.netty

netty-all

4.1.96.Final

io.netty

netty-buffer

4.1.96.Final

io.netty

netty-codec

4.1.96.Final

io.netty

netty-codec-http

4.1.96.Final

io.netty

netty-codec-http2

4.1.96.Final

io.netty

netty-codec-socks

4.1.96.Final

io.netty

netty-common

4.1.96.Final

io.netty

netty-handler

4.1.96.Final

io.netty

netty-handler-proxy

4.1.96.Final

io.netty

netty-resolver

4.1.96.Final

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-windows-x86_64

io.netty

netty-tcnative-classes

2.0.61.Final

io.netty

netty-transport

4.1.96.Final

io.netty

netty-transport-classes-epoll

4.1.96.Final

io.netty

netty-transport-classes-kqueue

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.96.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.10.11

org.apache.ant

ant-jsch

1.10.11

org.apache.ant

ant-launcher

1.10.11

org.apache.arrow

arrow-format

15.0.0

org.apache.arrow

arrow-memory-core

15.0.0

org.apache.arrow

arrow-memory-netty

15.0.0

org.apache.arrow

arrow-vector

15.0.0

org.apache.avro

avro

1.11.3

org.apache.avro

avro-ipc

1.11.3

org.apache.avro

avro-mapred

1.11.3

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.23.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.6

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.22.1

org.apache.logging.log4j

log4j-api

2.22.1

org.apache.logging.log4j

log4j-core

2.22.1

org.apache.logging.log4j

log4j-slf4j2-impl

2.22.1

org.apache.orc

orc-core

1.9.2-shaded-protobuf

org.apache.orc

orc-mapreduce

1.9.2-shaded-protobuf

org.apache.orc

orc-shims

1.9.2

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.ws.xmlschema

xmlschema-core

2.3.0

org.apache.xbean

xbean-asm9-shaded

4.23

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.31.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.collections

eclipse-collections

11.1.0

org.eclipse.collections

eclipse-collections-api

11.1.0

org.eclipse.jetty

jetty-client

9.4.52.v20230823

org.eclipse.jetty

jetty-continuation

9.4.52.v20230823

org.eclipse.jetty

jetty-http

9.4.52.v20230823

org.eclipse.jetty

jetty-io

9.4.52.v20230823

org.eclipse.jetty

jetty-jndi

9.4.52.v20230823

org.eclipse.jetty

jetty-plus

9.4.52.v20230823

org.eclipse.jetty

jetty-proxy

9.4.52.v20230823

org.eclipse.jetty

jetty-security

9.4.52.v20230823

org.eclipse.jetty

jetty-server

9.4.52.v20230823

org.eclipse.jetty

jetty-servlet

9.4.52.v20230823

org.eclipse.jetty

jetty-servlets

9.4.52.v20230823

org.eclipse.jetty

jetty-util

9.4.52.v20230823

org.eclipse.jetty

jetty-util-ajax

9.4.52.v20230823

org.eclipse.jetty

jetty-webapp

9.4.52.v20230823

org.eclipse.jetty

jetty-xml

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-api

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-client

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-common

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-server

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-servlet

9.4.52.v20230823

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.40

org.glassfish.jersey.containers

jersey-container-servlet-core

2.40

org.glassfish.jersey.core

jersey-client

2.40

org.glassfish.jersey.core

jersey-common

2.40

org.glassfish.jersey.core

jersey-server

2.40

org.glassfish.jersey.inject

jersey-hk2

2.40

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mlflow

mlflow-spark_2.12

2.9.1

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.6.1

org.roaringbitmap

RoaringBitmap

0.9.45

org.roaringbitmap

shims

0.9.45

org.rocksdb

rocksdbjni

8.3.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.11.0

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.7

org.slf4j

jul-to-slf4j

2.0.7

org.slf4j

slf4j-api

2.0.7

org.slf4j

slf4j-simple

1.7.25

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.1-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1