Databricks Runtime 15.1
The following release notes provide information about Databricks Runtime 15.1, powered by Apache Spark 3.5.0.
Databricks released this version in April 2024.
Tip
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
Breaking changes
Support for Databricks Runtime 15.0 and Databricks Runtime 15.0 for Machine Learning ends on May 31, 2024 because an upgrade of the pandas library from 1.5.3 to 2.0.3 in 15.0 caused a breaking change in multiple Apache Spark APIs. For details of the breaking changes, see The pandas Python library is downgraded to 1.5.3. To resolve this incompatibility, Databricks has released Databricks Runtime 15.1 and Databricks Runtime 15.1 for Machine Learning. The functionality supported by these releases is the same as the 15.0 releases, but with the pandas version downgraded to 1.5.3.
Any existing clusters or jobs that use Databricks Runtime 15.0 releases will continue to work but will not receive product or security updates. You can no longer configure new compute that uses Databricks Runtime 15.0 or Databricks Runtime 15.0 ML in the Databricks UI. If you are not using features or APIs that require pandas 2.0+, Databricks recommends switching those workloads to Databricks Runtime 15.1.
If you are using features or APIs available only in pandas 2.0+ and are not using any functions affected by this version, Databricks recommends switching to Databricks Runtime 15.1 and upgrading the default pandas package using cluster or notebook-scoped libraries. For example, to upgrade the pandas library in a Databricks notebook, use the following command:
%pip install pandas==2.0.3
The pandas Python library is downgraded to 1.5.3
In this release, the pandas library is downgraded from version 2.0.3 to 1.5.3. This downgrade was required because incompatibilities in the 2.0.3 version of pandas caused errors in some pandas API on Spark functions. The following lists the functions affected by the 2.0.3 version of pandas:
pyspark.date_range
: Theclosed
parameter is missing, causing errors when this function is called by existing code.The functionality of
GroupBy
withMultiIndex
columns is changed, causing unexpected behavior.Some
datetime
attributes ofDatetimeIndex
(for exampleday
,month
,year
, and so forth) returnint32
data types instead ofint64
data types.
The PySpark spark.sql()
function now fails for invalid argument types
In Databricks Runtime 15.1 and later, the args
parameter for named or positional parameters passed to the spark.sql()
function must be a dictionary or list. If any other object type is passed, the following error is raised: PySparkTypeError: [INVALID_TYPE] Argument `args` should not be a <arg_type>
.
Storing libraries in DBFS root is deprecated and disabled by default
To improve the security of libraries in a Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default starting with Databricks Runtime 15.1. Databricks recommends uploading all libraries, which includes Python libraries, JAR files, and Spark connectors, to workspace files, Unity Catalog volumes, or using a package repository. See Recommendations for uploading libraries.
To re-enable storing libraries in the DBFS root, set the following Spark configuration parameter: spark.databricks.driver.dbfsLibraryInstallationAllowed true
.
The default Python version is upgraded from 3.10 to 3.11
With Databricks Runtime 15.1, the default Python version is 3.11.0. For the list of upgraded Python libraries, see Library upgrades.
JDK 11 is removed
As previously announced, JDK 11 is removed from Databricks Runtime 15.1 and later. Databricks recommends upgrading to JDK 17 when upgrading to Databricks Runtime 15.1 and later. See Databricks SDK for Java.
The Python REPL now terminates gracefully
With Databricks Runtime 15.1, the notebook’s Python REPL process now terminates gracefully at exit. This causes any atexit hooks to be honored. For example, in a workflow with multiple Python notebook tasks, atexit hooks registered in the first task execute before the second Python notebook task runs.
New features and improvements
The * (star) clause is now supported in the WHERE clause
You can now use the star (*
) clause in the WHERE
clause to reference all columns from the SELECT
list.
For example, SELECT * FROM VALUES(1, 2) AS T(a1, a2) WHERE 1 IN(T.*)
.
The spark.sql.json.enablePartialResults
configuration is now enabled by default
The Spark configuration spark.sql.json.enablePartialResults
is now enabled by default, allowing the JSON parser to correctly handle parsing errors for complex types such as struct, map, and array without dropping the remaining fields. This change fixes a consistency issue for JSON data sources with column pruning.
Cluster libraries now support requirements.txt files
You can now install cluster libraries using a requirements.txt
file stored in a workspace file or Unity Catalog volume. In Single user and Shared access mode clusters, the requirements.txt
file can reference other files. In No isolation shared access mode clusters, only PyPI packages are supported. See Cluster libraries.
Databricks CLI now available in the web terminal
You can now use the Databricks CLI from the web terminal in the notebook. The notebook must be attached to a cluster in Single user or No isolation shared access mode. For details, see Use web terminal and Databricks CLI.
Add default Python repositories to cluster configurations
In Databricks Runtime 15.1 and later, you can configure global pip index-url
and extra-index-url
parameters for cluster and notebook-scoped library installation when configuring a cluster or defining a cluster policy. To do so, set the environment variables DATABRICKS_PIP_INDEX_URL
and DATABRICKS_PIP_EXTRA_INDEX_URL
.
Credential passthrough and Hive metastore table access controls are deprecated
Credential passthrough and Hive metastore table access controls are legacy data governance models. Upgrade to Unity Catalog to simplify the security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account. See What is Unity Catalog?.
Support for credential passthrough and Hive metastore table access controls will be removed in an upcoming DBR version.
Library upgrades
Upgraded Python libraries:
black from 22.6.0 to 23.3.0
boto3 from 1.24.28 to 1.34.39
botocore from 1.27.96 to 1.34.39
certifi from 2022.12.7 to 2023.7.22
cryptography from 39.0.1 to 41.0.3
databricks-sdk from 0.1.6 to 0.20.0
distlib from 0.3.7 to 0.3.8
googleapis-common-protos from 1.62.0 to 1.63.0
grpcio from 1.48.2 to 1.60.0
grpcio-status from 1.48.1 to 1.60.0
importlib-metadata from 4.6.4 to 6.0.0
ipykernel from 6.25.0 to 6.25.1
ipython from 8.14.0 to 8.15.0
ipywidgets from 7.7.2 to 8.0.4
jupyter_core from 5.2.0 to 5.3.0
jupyterlab-widgets from 1.0.0 to 3.0.5
matplotlib from 3.7.0 to 3.7.2
pip from 22.3.1 to 23.2.1
platformdirs from 2.5.2 to 3.10.0
protobuf from 4.24.0 to 4.24.1
pyarrow from 8.0.0 to 14.0.1
Pygments from 2.11.2 to 2.15.1
pyodbc from 4.0.32 to 4.0.38
requests from 2.28.1 to 2.31.0
s3transfer from 0.6.2 to 0.10.0
scikit-learn from 1.1.1 to 1.3.0
scipy from 1.10.0 to 1.11.1
setuptools from 65.6.3 to 68.0.0
statsmodels from 0.13.5 to 0.14.0
tenacity from 8.1.0 to 8.2.2
tornado from 6.1 to 6.3.2
typing_extensions from 4.4.0 to 4.7.1
urllib3 from 1.26.14 to 1.26.16
virtualenv from 20.16.7 to 20.24.2
widgetsnbextension from 3.6.1 to 4.0.5
zipp from 1.0.0 to 3.11.0
Upgraded R libraries:
arrow from 12.0.1 to 14.0.0.2
askpass from 1.1 to 1.2.0
base from 4.3.1 to 4.3.2
brew from 1.0-8 to 1.0-10
brio from 1.1.3 to 1.1.4
bslib from 0.5.0 to 0.6.1
cli from 3.6.1 to 3.6.2
commonmark from 1.9.0 to 1.9.1
compiler from 4.3.1 to 4.3.2
config from 0.3.1 to 0.3.2
cpp11 from 0.4.4 to 0.4.7
credentials from 1.3.2 to 2.0.1
curl from 5.0.1 to 5.2.0
data.table from 1.14.8 to 1.15.0
datasets from 4.3.1 to 4.3.2
DBI from 1.1.3 to 1.2.1
dbplyr from 2.3.3 to 2.4.0
desc from 1.4.2 to 1.4.3
digest from 0.6.33 to 0.6.34
dplyr from 1.1.2 to 1.1.4
e1071 from 1.7-13 to 1.7-14
evaluate from 0.21 to 0.23
fansi from 1.0.4 to 1.0.6
fontawesome from 0.5.1 to 0.5.2
fs from 1.6.2 to 1.6.3
future from 1.33.0 to 1.33.1
future.apply from 1.11.0 to 1.11.1
gargle from 1.5.1 to 1.5.2
gert from 1.9.2 to 2.0.1
ggplot2 from 3.4.2 to 3.4.4
glmnet from 4.1-7 to 4.1-8
glue from 1.6.2 to 1.7.0
graphics from 4.3.1 to 4.3.2
grDevices from 4.3.1 to 4.3.2
grid from 4.3.1 to 4.3.2
gtable from 0.3.3 to 0.3.4
hardhat from 1.3.0 to 1.3.1
haven from 2.5.3 to 2.5.4
htmltools from 0.5.5 to 0.5.7
htmlwidgets from 1.6.2 to 1.6.4
httpuv from 1.6.11 to 1.6.14
httr from 1.4.6 to 1.4.7
httr2 from 0.2.3 to 1.0.0
jsonlite from 1.8.7 to 1.8.8
knitr from 1.43 to 1.45
labeling from 0.4.2 to 0.4.3
later from 1.3.1 to 1.3.2
lava from 1.7.2.1 to 1.7.3
lifecycle from 1.0.3 to 1.0.4
listenv from 0.9.0 to 0.9.1
lubridate from 1.9.2 to 1.9.3
markdown from 1.7 to 1.12
methods from 4.3.1 to 4.3.2
openssl from 2.0.6 to 2.1.1
parallel from 4.3.1 to 4.3.2
pkgbuild from 1.4.2 to 1.4.3
pkgload from 1.3.2.1 to 1.3.4
plyr from 1.8.8 to 1.8.9
prettyunits from 1.1.1 to 1.2.0
pROC from 1.18.4 to 1.18.5
processx from 3.8.2 to 3.8.3
prodlim from 2023.03.31 to 2023.08.28
progress from 1.2.2 to 1.2.3
progressr from 0.13.0 to 0.14.0
promises from 1.2.0.1 to 1.2.1
ps from 1.7.5 to 1.7.6
purrr from 1.0.1 to 1.0.2
ragg from 1.2.5 to 1.2.7
Rcpp from 1.0.11 to 1.0.12
RcppEigen from 0.3.3.9.3 to 0.3.3.9.4
readr from 2.1.4 to 2.1.5
recipes from 1.0.6 to 1.0.9
rematch from 1.0.1 to 2.0.0
remotes from 2.4.2 to 2.4.2.1
reprex from 2.0.2 to 2.1.0
rlang from 1.1.1 to 1.1.3
rmarkdown from 2.23 to 2.25
RODBC from 1.3-20 to 1.3-23
roxygen2 from 7.2.3 to 7.3.1
rprojroot from 2.0.3 to 2.0.4
Rserve from 1.8-11 to 1.8-13
RSQLite from 2.3.1 to 2.3.5
sass from 0.4.6 to 0.4.8
scales from 1.2.1 to 1.3.0
shiny from 1.7.4.1 to 1.8.0
sparklyr from 1.8.1 to 1.8.4
splines from 4.3.1 to 4.3.2
stats from 4.3.1 to 4.3.2
stats4 from 4.3.1 to 4.3.2
stringi from 1.7.12 to 1.8.3
stringr from 1.5.0 to 1.5.1
systemfonts from 1.0.4 to 1.0.5
tcltk from 4.3.1 to 4.3.2
testthat from 3.1.10 to 3.2.1
textshaping from 0.3.6 to 0.3.7
tidyr from 1.3.0 to 1.3.1
timechange from 0.2.0 to 0.3.0
timeDate from 4022.108 to 4032.109
tinytex from 0.45 to 0.49
tools from 4.3.1 to 4.3.2
utf8 from 1.2.3 to 1.2.4
utils from 4.3.1 to 4.3.2
uuid from 1.1-0 to 1.2-0
vctrs from 0.6.3 to 0.6.5
vroom from 1.6.3 to 1.6.5
waldo from 0.5.1 to 0.5.2
withr from 2.5.0 to 3.0.0
xfun from 0.39 to 0.41
xml2 from 1.3.5 to 1.3.6
yaml from 2.3.7 to 2.3.8
zip from 2.3.0 to 2.3.1
Upgraded Java libraries:
com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.15.1 to 2.16.0
com.google.flatbuffers.flatbuffers-java from 1.12.0 to 23.5.26
com.typesafe.config from 1.2.1 to 1.4.3
org.apache.ant.ant from 1.9.16 to 1.10.11
org.apache.ant.ant-jsch from 1.9.16 to 1.10.11
org.apache.ant.ant-launcher from 1.9.16 to 1.10.11
org.apache.arrow.arrow-format from 12.0.1 to 15.0.0
org.apache.arrow.arrow-memory-core from 12.0.1 to 15.0.0
org.apache.arrow.arrow-memory-netty from 12.0.1 to 15.0.0
org.apache.arrow.arrow-vector from 12.0.1 to 15.0.0
org.apache.avro.avro from 1.11.2 to 1.11.3
org.apache.avro.avro-ipc from 1.11.2 to 1.11.3
org.apache.avro.avro-mapred from 1.11.2 to 1.11.3
org.apache.logging.log4j.log4j-1.2-api from 2.20.0 to 2.22.1
org.apache.logging.log4j.log4j-api from 2.20.0 to 2.22.1
org.apache.logging.log4j.log4j-core from 2.20.0 to 2.22.1
org.apache.logging.log4j.log4j-slf4j2-impl from 2.20.0 to 2.22.1
org.postgresql.postgresql from 42.6.0 to 42.6.1
Apache Spark
Databricks Runtime 15.1 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.3 LTS, as well as the following additional bug fixes and improvements made to Spark:
[SPARK-45527] [DBRRM-805][ES-1073714] Revert “[SC-151626][CORE] Use fraction to do the resource calculation”
[SPARK-47102] [DBRRM-803][SC-158253][SQL] Add the
COLLATION_ENABLED
config flag[SPARK-46973] [SC-158140][DBRRM-777][SQL] Skip V2 table lookup when a table is in V1 table cache
[SPARK-46808] [SC-154113][PYTHON] Refine error classes in Python with automatic sorting function
[SPARK-47251] [SC-158291][PYTHON] Block invalid types from the
args
argument forsql
command[SPARK-47251] [SC-158121][PYTHON] Block invalid types from the
args
argument forsql
command[SPARK-47108] [SC-157204][CORE] Set
derby.connection.requireAuthentication
tofalse
explicitly in CLIs[SPARK-45599] [SC-157537][CORE] Use object equality in OpenHashSet
[SPARK-47099] [SC-157352][SQL] Use
ordinalNumber
to uniformly set the value ofparamIndex
for the error classUNEXPECTED_INPUT_TYPE
[SPARK-46558] [SC-151363][CONNECT] Extract a helper function to eliminate the duplicate code that retrieves
MessageParameters
fromErrorParams
inGrpcExceptionConverter
[SPARK-43117] [SC-156177][CONNECT] Make
ProtoUtils.abbreviate
support repeated fields[SPARK-46342] [SC-150283][SQL] Replace
IllegalStateException
bySparkException.internalError
in sql[SPARK-47123] [SC-157412][CORE] JDBCRDD does not correctly handle errors in getQueryOutputSchema
[SPARK-47189] [SC-157667][SQL] Tweak column error names and text
[SPARK-45789] [SC-157101][SQL] Support DESCRIBE TABLE for clustering columns
[SPARK-43256] [SC-157699][SQL] Remove error class LEGACYERROR_TEMP_2021
[SPARK-47201] [SC-157782][PYTHON][CONNECT]
sameSemantics
checks input types[SPARK-47183] [SC-157660][PYTHON] Fix the error class for
sameSemantics
[SPARK-47179] [SC-157663][SQL] Improve error message from SparkThrowableSuite for better debuggability
[SPARK-46965] [SC-155791][CORE] Check
logType
inUtils.getLog
[SPARK-46832] [SC-156774][SQL] Introducing Collate and Collation expressions
[SPARK-46946] [SC-155721][SQL] Supporting broadcast of multiple filtering keys in DynamicPruning
[SPARK-47079] [VARIANT-22][SC-157598][PYTHON][SQL][CONNECT] Add Variant type info to PySpark
[SPARK-47101] [SC-157360][SQL] Allow comma to be used in top-level column names and remove check nested type definition in
HiveExternalCatalog.verifyDataSchema
[SPARK-47173] [SC-157571][SS][UI] Fix a typo in streaming UI explanation
[SPARK-47113] [SC-157348][CORE] Revert S3A endpoint fixup logic of SPARK-35878
[SPARK-47130] [SC-157462][CORE] Use listStatus to bypass block location info when cleaning driver logs
[SPARK-43259] [SC-157368][SQL] Assign a name to the error class LEGACYERROR_TEMP_2024
[SPARK-47104] [SC-157355][SQL]
TakeOrderedAndProjectExec
should initialize the unsafe projection[SPARK-47137] [SC-157463][PYTHON][CONNECT] Add getAll to spark.conf for feature parity with Scala
[SPARK-46924] [SC-154905][CORE] Fix
Load New
button inMaster/HistoryServer
Log UI[SPARK-47069] [SC-157370][PYTHON][CONNECT] Introduce
spark.profile.show/dump
for SparkSession-based profiling[SPARK-46812] [SC-157075][SQL][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile
[SPARK-46833] [SC-155866][SQL] Collations - Introducing CollationFactory which provides comparison and hashing rules for supported collations
[SPARK-47057] [SC-157036][PYTHON] Reeanble MyPy data test
[SPARK-46934] [SC-157084][SQL] Read/write roundtrip for struct type with special characters with HMS
[SPARK-46727] [SC-153472][SQL] Port
classifyException()
in JDBC dialects on error classes[SPARK-46964] [SC-155740][SQL] Change the signature of the hllInvalidLgK query execution error to take an integer as 4th argument
[SPARK-46949] [SC-155719][SQL] Support CHAR/VARCHAR through ResolveDefaultColumns
[SPARK-46972] [SC-155888][SQL] Fix asymmetrical replacement for char/varchar in V2SessionCatalog.createTable
[SPARK-47014] [SC-156778][PYTHON][CONNECT] Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession
[SPARK-46689] [SC-156221][SPARK-46690][PYTHON][CONNECT] Support v2 profiling in group/cogroup applyInPandas/applyInArrow
[SPARK-47040] [SC-156773][CONNECT] Allow Spark Connect Server Script to wait
[SPARK-46688] [SC-156113][SPARK-46691][PYTHON][CONNECT] Support v2 profiling in aggregate Pandas UDFs
[SPARK-47004] [SC-156499][CONNECT] Added more tests to ClientStreamingQuerySuite to increase Scala client test coverage
[SPARK-46994] [SC-156169][PYTHON] Refactor PythonWrite to prepare for supporting python data source streaming write
[SPARK-46979] [SC-156587][SS] Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider
[SPARK-46960] [SC-155940][SS] Testing Multiple Input Streams with TransformWithState operator
[SPARK-46966] [SC-156112][Python] Add UDTF API for ‘analyze’ method to indicate subset of input table columns to select
[SPARK-46895] [SC-155950][CORE] Replace Timer with single thread scheduled executor
[SPARK-46967] [SC-155815][CORE][UI] Hide
Thread Dump
andHeap Histogram
ofDead
executors inExecutors
UI[SPARK-46512] [SC-155826][CORE] Optimize shuffle reading when both sort and combine are used.
[SPARK-46958] [SC-155942][SQL] Add missing timezone to coerce default values
[SPARK-46754] [SC-153830][SQL][AVRO] Fix compression code resolution in avro table definition and write options
[SPARK-46728] [SC-154207][PYTHON] Check Pandas installation properly
[SPARK-43403] [SC-154216][UI] Ensure old SparkUI in HistoryServer has been detached before loading new one
[SPARK-39910] [SC-156203][SQL] Delegate path qualification to filesystem during DataSource file path globbing
[SPARK-47000] [SC-156123] Revert “[SC-156123][CORE] Use
getTotalMemorySize
…[SPARK-46819] [SC-154560][CORE] Move error categories and states into JSON
[SPARK-46997] [SC-156175][CORE] Enable
spark.worker.cleanup.enabled
by default[SPARK-46865] [SC-156168][SS] Add Batch Support for TransformWithState Operator
[SPARK-46987] [SC-156040][CONNECT]
ProtoUtils.abbreviate
avoid unnecessarysetField
operation[SPARK-46970] [SC-155816][CORE] Rewrite
OpenHashSet#hasher
withpattern matching
[SPARK-46984] [SC-156024][PYTHON] Remove pyspark.copy_func
[SPARK-47000] [SC-156123][CORE] Use
getTotalMemorySize
inWorkerArguments
[SPARK-46980] [SC-155914][SQL][MINOR] Avoid using internal APIs in dataframe end-to-end tests
[SPARK-46931] Revert “[SC-155661][PS] Implement
{Frame, Series}.to_hdf
”[SPARK-46618] [SC-153828][SQL] Improve error messages for DATA_SOURCE_NOT_FOUND error
[SPARK-46849] [SC-154586][SQL] Run optimizer on CREATE TABLE column defaults
[SPARK-46911] [SC-155724][SS] Adding deleteIfExists operator to StatefulProcessorHandleImpl
[SPARK-43273] [SQL] Support
lz4raw
compression codec for Parquet[SPARK-47002] [SC-156223][Python] Return better error message if UDTF ‘analyze’ method ‘orderBy’ field accidentally returns a list of strings
[SPARK-46890] [SC-155802][SQL] Fix CSV parsing bug with existence default values and column pruning
[SPARK-46977] [SC-155892][CORE] A failed request to obtain a token from one NameNode should not skip subsequent token requests
[SPARK-46608] [SC-151806][SQL] Restore backward compatibility of
JdbcDialect.classifyException
[SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause
[SPARK-46908] [SC-155702][SQL] Support star clause in WHERE clause
[SPARK-46852] [SC-155665][SS] Remove use of explicit key encoder and pass it implicitly to the operator for transformWithState operator
[SPARK-46864] [SC-155669][SS] Onboard Arbitrary StateV2 onto New Error Class Framework
[SPARK-45807] [SC-155706][SQL] Return View after calling replaceView(..)
[SPARK-46899] [SC-154651][CORE] Remove
POST
APIs fromMasterWebUI
whenspark.ui.killEnabled
isfalse
[SPARK-46868] [SC-154539][CORE] Support Spark Worker Log UI
[SPARK-46931] [SC-155661][PS] Implement
{Frame, Series}.to_hdf
[SPARK-46940] [SC-155667][CORE] Remove unused
updateSparkConfigFromProperties
andisAbsoluteURI
ino.a.s.u.Utils
[SPARK-46929] [SC-155659][CORE][CONNECT][SS] Use ThreadUtils.shutdown to close thread pools
[SPARK-46400] [SC-155658][CORE][SQL] When there are corrupted files in the local maven repo, skip this cache and try again
[SPARK-46932] [SC-155655] Clean up the imports in
pyspark.pandas.test_*
[SPARK-46683] [SC-154120] Write a subquery generator that generates subqueries permutations to increase testing coverage
[SPARK-46914] [SC-154776][UI] Shorten app name in the summary table on the History Page
[SPARK-46831] [SC-154699][SQL] Collations - Extending StringType and PhysicalStringType with collationId field
[SPARK-46816] [SC-154415][SS] Add base support for new arbitrary state management operator, single valueState type, multiple state variables and underlying support for column families for RocksDBStateStoreProvider with/without changelog checkpointing
[SPARK-46925] [SC-154890][PYTHON][CONNECT] Add a warning that instructs to install memory_profiler for memory profiling
[SPARK-46927] [SC-154904][PYTHON] Make
assertDataFrameEqual
work properly without PyArrow[SPARK-46021] [SC-148987][CORE] Support cancel future jobs belonging to a job group
[SPARK-46747] [SC-154789][SQL] Avoid scan in getTableExistsQuery for JDBC Dialects
[SPARK-46904] [SC-154704][UI] Fix display issue of History UI summary
[SPARK-46870] [SC-154541][CORE] Support Spark Master Log UI
[SPARK-46893] [SC-154780][UI] Remove inline scripts from UI descriptions
[SPARK-46910] [SC-154760][PYTHON] Eliminate JDK Requirement in PySpark Installation
[SPARK-46907] [SC-154736][CORE] Show driver log location in Spark History Server
[SPARK-46902] [SC-154661][UI] Fix Spark History Server UI for using un-exported setAppLimit
[SPARK-46687] [SC-154725][PYTHON][CONNECT] Basic support of SparkSession-based memory profiler
[SPARK-46623] [SC-153831][CORE][MLLIB][SQL] Replace SimpleDateFormat with DateTimeFormatter
[SPARK-46875] [SC-154627][SQL] When the
mode
is null, aNullPointException
shouldnot
be thrown[SPARK-46872] [SC-154543][CORE] Recover
log-view.js
to be non-module[SPARK-46883] [SC-154629][CORE] Support
/json/clusterutilization
API[SPARK-46886] [SC-154630][CORE] Enable
spark.ui.prometheus.enabled
by default[SPARK-46873] [SC-154638][SS] Do not recreate new StreamingQueryManager for the same Spark Session
[SPARK-46829] [SC-154419][CORE] Remove createExecutorEnv from SparkEnv
[SPARK-46827] [SC-154280][CORE] Make
RocksDBPersistenceEngine
to support a symbolic link[SPARK-46903] [SC-154662][CORE] Support Spark History Server Log UI
[SPARK-46874] [SC-154646][PYTHON] Remove
pyspark.pandas
dependency fromassertDataFrameEqual
[SPARK-46889] [SC-154639][CORE] Validate
spark.master.ui.decommission.allow.mode
setting[SPARK-46850] [SC-154443][SQL] Convert
_LEGACY_ERROR_TEMP_2102
toUNSUPPORTED_DATATYPE
[SPARK-46704] [SC-153323][CORE][UI] Fix
MasterPage
to sortRunning Drivers
table byDuration
column correctly[SPARK-46796] [SC-154279][SS] Ensure the correct remote files (mentioned in metadata.zip) are used on RocksDB version load
[SPARK-46888] [SC-154636][CORE] Fix
Master
to reject/workers/kill/
requests if decommission is disabled[SPARK-46818] [SC-154422][SQL] Improve error messages for Range with non-foldable inputs
[SPARK-46898] [SC-154649][CONNECT] Simplify the protobuf function transformation in Planner
[SPARK-46828] [SC-154413][SQL] Remove the invalid assertion of remote mode for spark sql shell
[SPARK-46733] [SC-154274][CORE] Simplify the BlockManager by the exit operation only depend on interrupt thread.
[SPARK-46777] [SC-154016][SS] Refactor
StreamingDataSourceV2Relation
catalyst structure to be more on-par with the batch version[SPARK-46515] [SC-151716] Add MONTHNAME function
[SPARK-46823] [SC-154276][CONNECT][PYTHON]
LocalDataToArrowConversion
should check the nullability[SPARK-46787] [SC-154404][CONNECT]
bloomFilter
function should throwAnalysisException
for invalid input[SPARK-46779] [SC-154045][SQL]
InMemoryRelation
instances of the same cached plan should be semantically equivalent[SPARK-45827] [SC-153973] Disallow partitioning on Variant column
[SPARK-46797] [SC-154085][CORE] Rename
spark.deploy.spreadOut
tospark.deploy.spreadOutApps
[SPARK-46094] [SC-153421] Support Executor JVM Profiling
[SPARK-46746] [SC-153557][SQL][AVRO] Attach codec extension to avro datasource files
[SPARK-46698] [SC-153288][CORE] Replace Timer with single thread scheduled executor for ConsoleProgressBar.
[SPARK-46629] [SC-153835] Fix for STRUCT type DDL not picking up nullability and comment
[SPARK-46750] [SC-153772][CONNECT][PYTHON] DataFrame APIs code clean up
[SPARK-46769] [SC-153979][SQL] Refine timestamp related schema inference
[SPARK-46765] [SC-153904][PYTHON][CONNECT] Make
shuffle
specify the datatype ofseed
[SPARK-46579] [SC-151481][SQL] Redact JDBC url in errors and logs
[SPARK-46686] [SC-153875][PYTHON][CONNECT] Basic support of SparkSession based Python UDF profiler
[SPARK-46748] Revert “[SC-153800][CORE] Remove
*slav**.sh
scri…[SPARK-46707] [SC-153833][SQL] Added throwable field to expressions to improve predicate pushdown
[SPARK-46519] [SC-151278][SQL] Clear unused error classes from
error-classes.json
file[SPARK-46677] [SC-153426][SQL][CONNECT] Fix
dataframe["*"]
resolution[SPARK-46317] [SC-150184][PYTHON][CONNECT] Match minor behaviour matching in SparkSession with full test coverage
[SPARK-46748] [SC-153800][CORE] Remove
*slav**.sh
scripts[SPARK-46663] [SC-153490][PYTHON] Disable memory profiler for pandas UDFs with iterators
[SPARK-46410] [SC-150776][SQL] Assign error classes/subclasses to JdbcUtils.classifyException
[SPARK-46277] [SC-150126][PYTHON] Validate startup urls with the config being set
[SPARK-46612] [SC-153536][SQL] Do not convert array type string retrieved from jdbc driver
[SPARK-46254] [SC-149788][PYTHON] Remove stale Python 3.8/3.7 version checking
[SPARK-46490] [SC-151335][SQL] Require error classes in
SparkThrowable
sub-classes[SPARK-46383] [SC-153274][SC-147443][WARMFIX] Reduce Driver Heap Usage by Reducing the Lifespan of
TaskInfo.accumulables()
[SPARK-46541] [SC-153109][SQL][CONNECT] Fix the ambiguous column reference in self join
[SPARK-46381] [SC-150495][SQL] Migrate sub-classes of
AnalysisException
to error classes[SPARK-46351] [SC-150335][SQL] Require an error class in
AnalysisException
[SPARK-46220] [SC-149689][SQL] Restrict charsets in
decode()
[SPARK-46369] [SC-150340][CORE] Remove
kill
link fromRELAUNCHING
drivers inMasterPage
[SPARK-46052] [SC-153284][CORE] Remove function TaskScheduler.killAllTaskAttempts
[SPARK-46536] [SC-153164][SQL] Support GROUP BY calendar_interval_type
[SPARK-46675] [SC-153209][SQL] Remove unused inferTimestampNTZ in ParquetReadSupport
[SPARK-46717] [SC-153423][CORE] Simplify
ReloadingX509TrustManager
by the exit operation only depend on interrupt thread.[SPARK-46722] [SC-153438][CONNECT] Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)
[SPARK-46187] [SC-149580][SQL] Align codegen and non-codegen implementation of
StringDecode
[SPARK-46258] [SC-149799][CORE] Add
RocksDBPersistenceEngine
[SPARK-46216] [SC-149676][CORE] Improve
FileSystemPersistenceEngine
to support compressions[SPARK-46189] [SC-149567][PS][SQL] Perform comparisons and arithmetic between same types in various Pandas aggregate functions to avoid interpreted mode errors
[SPARK-46184] [SC-149563][CORE][SQL][CONNECT][MLLIB] Reduce stack depth by replace Option.isDefined with Option.isEmpty
[SPARK-46700] [SC-153329][CORE] Count the last spilling for the shuffle disk spilling bytes metric
[SPARK-45642] [SC-150286][CORE][SQL] Fix
FileSystem.isFile & FileSystem.isDirectory is deprecated
[SPARK-46640] [SC-153272][SQL] Fix RemoveRedundantAlias by excluding subquery attributes
[SPARK-46681] [SC-153287][CORE] Refactor
ExecutorFailureTracker#maxNumExecutorFailures
to avoid calculatingdefaultMaxNumExecutorFailures
whenMAX_EXECUTOR_FAILURES
is configured[SPARK-46695] [SC-153289][SQL][HIVE] Always setting hive.execution.engine to mr
[SPARK-46325] [SC-150228][CONNECT] Remove unnecessary override functions when constructing
WrappedCloseableIterator
inResponseValidator#wrapIterator
[SPARK-46232] [SC-149699][PYTHON] Migrate all remaining ValueError into PySpark error framework.
[SPARK-46547] [SC-153174][SS] Swallow non-fatal exception in maintenance task to avoid deadlock between maintenance thread and streaming aggregation operator
[SPARK-46169] [SC-149373][PS] Assign appropriate JIRA numbers for missing parameters from
DataFrame
API.[SPARK-45857] [SC-148096][SQL] Enforce the error classes in sub-classes of
AnalysisException
Databricks ODBC/JDBC driver support
Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).
System environment
Operating System: Ubuntu 22.04.4 LTS
Java: Zulu 8.74.0.17-CA-linux64
Scala: 2.12.15
Python: 3.11.0
R: 4.3.2
Delta Lake: 3.1.0
Installed Python libraries
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
asttokens |
2.0.5 |
astunparse |
1.6.3 |
azure-core |
1.30.1 |
azure-storage-blob |
12.19.1 |
azure-storage-file-datalake |
12.14.0 |
backcall |
0.2.0 |
black |
23.3.0 |
blinker |
1.4 |
boto3 |
1.34.39 |
botocore |
1.34.39 |
cachetools |
5.3.3 |
certifi |
2023.7.22 |
cffi |
1.15.1 |
chardet |
4.0.0 |
charset-normalizer |
2.0.4 |
click |
8.0.4 |
cloudpickle |
2.2.1 |
comm |
0.1.2 |
contourpy |
1.0.5 |
cryptography |
41.0.3 |
cycler |
0.11.0 |
Cython |
0.29.32 |
databricks-sdk |
0.20.0 |
dbus-python |
1.2.18 |
debugpy |
1.6.7 |
decorator |
5.1.1 |
distlib |
0.3.8 |
entrypoints |
0.4 |
executing |
0.8.3 |
facets-overview |
1.1.1 |
filelock |
3.13.1 |
fonttools |
4.25.0 |
gitdb |
4.0.11 |
GitPython |
3.1.42 |
google-api-core |
2.17.1 |
google-auth |
2.28.2 |
google-cloud-core |
2.4.1 |
google-cloud-storage |
2.15.0 |
google-crc32c |
1.5.0 |
google-resumable-media |
2.7.0 |
googleapis-common-protos |
1.63.0 |
grpcio |
1.60.0 |
grpcio-status |
1.60.0 |
httplib2 |
0.20.2 |
idna |
3.4 |
importlib-metadata |
6.0.0 |
ipyflow-core |
0.0.198 |
ipykernel |
6.25.1 |
ipython |
8.15.0 |
ipywidgets |
8.0.4 |
isodate |
0.6.1 |
jedi |
0.18.1 |
jeepney |
0.7.1 |
jmespath |
0.10.0 |
joblib |
1.2.0 |
jupyter_client |
7.4.9 |
jupyter_core |
5.3.0 |
jupyterlab-widgets |
3.0.5 |
keyring |
23.5.0 |
kiwisolver |
1.4.4 |
launchpadlib |
1.10.16 |
lazr.restfulclient |
0.14.4 |
lazr.uri |
1.0.6 |
matplotlib |
3.7.2 |
matplotlib-inline |
0.1.6 |
mlflow-skinny |
2.11.1 |
more-itertools |
8.10.0 |
mypy-extensions |
0.4.3 |
nest-asyncio |
1.5.6 |
numpy |
1.23.5 |
oauthlib |
3.2.0 |
packaging |
23.2 |
pandas |
1.5.3 |
parso |
0.8.3 |
pathspec |
0.10.3 |
patsy |
0.5.3 |
pexpect |
4.8.0 |
pickleshare |
0.7.5 |
Pillow |
9.4.0 |
pip |
23.2.1 |
platformdirs |
3.10.0 |
plotly |
5.9.0 |
prompt-toolkit |
3.0.36 |
protobuf |
4.24.1 |
psutil |
5.9.0 |
psycopg2 |
2.9.3 |
ptyprocess |
0.7.0 |
pure-eval |
0.2.2 |
pyarrow |
14.0.1 |
pyasn1 |
0.4.8 |
pyasn1-modules |
0.2.8 |
pyccolo |
0.0.52 |
pycparser |
2.21 |
pydantic |
1.10.6 |
Pygments |
2.15.1 |
PyGObject |
3.42.1 |
PyJWT |
2.3.0 |
pyodbc |
4.0.38 |
pyparsing |
3.0.9 |
python-dateutil |
2.8.2 |
python-lsp-jsonrpc |
1.1.1 |
pytz |
2022.7 |
PyYAML |
6.0 |
pyzmq |
23.2.0 |
requests |
2.31.0 |
rsa |
4.9 |
s3transfer |
0.10.0 |
scikit-learn |
1.3.0 |
scipy |
1.11.1 |
seaborn |
0.12.2 |
SecretStorage |
3.3.1 |
setuptools |
68.0.0 |
six |
1.16.0 |
smmap |
5.0.1 |
sqlparse |
0.4.4 |
ssh-import-id |
5.11 |
stack-data |
0.2.0 |
statsmodels |
0.14.0 |
tenacity |
8.2.2 |
threadpoolctl |
2.2.0 |
tokenize-rt |
4.2.1 |
tornado |
6.3.2 |
traitlets |
5.7.1 |
typing_extensions |
4.7.1 |
tzdata |
2022.1 |
ujson |
5.4.0 |
unattended-upgrades |
0.1 |
urllib3 |
1.26.16 |
virtualenv |
20.24.2 |
wadllib |
1.3.6 |
wcwidth |
0.2.5 |
wheel |
0.38.4 |
widgetsnbextension |
4.0.5 |
zipp |
3.11.0 |
Installed R libraries
R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
arrow |
14.0.0.2 |
askpass |
1.2.0 |
assertthat |
0.2.1 |
backports |
1.4.1 |
base |
4.3.2 |
base64enc |
0.1-3 |
bigD |
0.2.0 |
bit |
4.0.5 |
bit64 |
4.0.5 |
bitops |
1.0-7 |
blob |
1.2.4 |
boot |
1.3-28 |
brew |
1.0-10 |
brio |
1.1.4 |
broom |
1.0.5 |
bslib |
0.6.1 |
cachem |
1.0.8 |
callr |
3.7.3 |
caret |
6.0-94 |
cellranger |
1.1.0 |
chron |
2.3-61 |
class |
7.3-22 |
cli |
3.6.2 |
clipr |
0.8.0 |
clock |
0.7.0 |
cluster |
2.1.4 |
codetools |
0.2-19 |
colorspace |
2.1-0 |
commonmark |
1.9.1 |
compiler |
4.3.2 |
config |
0.3.2 |
conflicted |
1.2.0 |
cpp11 |
0.4.7 |
crayon |
1.5.2 |
credentials |
2.0.1 |
curl |
5.2.0 |
data.table |
1.15.0 |
datasets |
4.3.2 |
DBI |
1.2.1 |
dbplyr |
2.4.0 |
desc |
1.4.3 |
devtools |
2.4.5 |
diagram |
1.6.5 |
diffobj |
0.3.5 |
digest |
0.6.34 |
downlit |
0.4.3 |
dplyr |
1.1.4 |
dtplyr |
1.3.1 |
e1071 |
1.7-14 |
ellipsis |
0.3.2 |
evaluate |
0.23 |
fansi |
1.0.6 |
farver |
2.1.1 |
fastmap |
1.1.1 |
fontawesome |
0.5.2 |
forcats |
1.0.0 |
foreach |
1.5.2 |
foreign |
0.8-85 |
forge |
0.2.0 |
fs |
1.6.3 |
future |
1.33.1 |
future.apply |
1.11.1 |
gargle |
1.5.2 |
generics |
0.1.3 |
gert |
2.0.1 |
ggplot2 |
3.4.4 |
gh |
1.4.0 |
git2r |
0.33.0 |
gitcreds |
0.1.2 |
glmnet |
4.1-8 |
globals |
0.16.2 |
glue |
1.7.0 |
googledrive |
2.1.1 |
googlesheets4 |
1.1.1 |
gower |
1.0.1 |
graphics |
4.3.2 |
grDevices |
4.3.2 |
grid |
4.3.2 |
gridExtra |
2.3 |
gsubfn |
0.7 |
gt |
0.10.1 |
gtable |
0.3.4 |
hardhat |
1.3.1 |
haven |
2.5.4 |
highr |
0.10 |
hms |
1.1.3 |
htmltools |
0.5.7 |
htmlwidgets |
1.6.4 |
httpuv |
1.6.14 |
httr |
1.4.7 |
httr2 |
1.0.0 |
ids |
1.0.1 |
ini |
0.3.1 |
ipred |
0.9-14 |
isoband |
0.2.7 |
iterators |
1.0.14 |
jquerylib |
0.1.4 |
jsonlite |
1.8.8 |
juicyjuice |
0.1.0 |
KernSmooth |
2.23-21 |
knitr |
1.45 |
labeling |
0.4.3 |
later |
1.3.2 |
lattice |
0.21-8 |
lava |
1.7.3 |
lifecycle |
1.0.4 |
listenv |
0.9.1 |
lubridate |
1.9.3 |
magrittr |
2.0.3 |
markdown |
1.12 |
MASS |
7.3-60 |
Matrix |
1.5-4.1 |
memoise |
2.0.1 |
methods |
4.3.2 |
mgcv |
1.8-42 |
mime |
0.12 |
miniUI |
0.1.1.1 |
mlflow |
2.10.0 |
ModelMetrics |
1.2.2.2 |
modelr |
0.1.11 |
munsell |
0.5.0 |
nlme |
3.1-163 |
nnet |
7.3-19 |
numDeriv |
2016.8-1.1 |
openssl |
2.1.1 |
parallel |
4.3.2 |
parallelly |
1.36.0 |
pillar |
1.9.0 |
pkgbuild |
1.4.3 |
pkgconfig |
2.0.3 |
pkgdown |
2.0.7 |
pkgload |
1.3.4 |
plogr |
0.2.0 |
plyr |
1.8.9 |
praise |
1.0.0 |
prettyunits |
1.2.0 |
pROC |
1.18.5 |
processx |
3.8.3 |
prodlim |
2023.08.28 |
profvis |
0.3.8 |
progress |
1.2.3 |
progressr |
0.14.0 |
promises |
1.2.1 |
proto |
1.0.0 |
proxy |
0.4-27 |
ps |
1.7.6 |
purrr |
1.0.2 |
R6 |
2.5.1 |
ragg |
1.2.7 |
randomForest |
4.7-1.1 |
rappdirs |
0.3.3 |
rcmdcheck |
1.4.0 |
RColorBrewer |
1.1-3 |
Rcpp |
1.0.12 |
RcppEigen |
0.3.3.9.4 |
reactable |
0.4.4 |
reactR |
0.5.0 |
readr |
2.1.5 |
readxl |
1.4.3 |
recipes |
1.0.9 |
rematch |
2.0.0 |
rematch2 |
2.1.2 |
remotes |
2.4.2.1 |
reprex |
2.1.0 |
reshape2 |
1.4.4 |
rlang |
1.1.3 |
rmarkdown |
2.25 |
RODBC |
1.3-23 |
roxygen2 |
7.3.1 |
rpart |
4.1.21 |
rprojroot |
2.0.4 |
Rserve |
1.8-13 |
RSQLite |
2.3.5 |
rstudioapi |
0.15.0 |
rversions |
2.1.2 |
rvest |
1.0.3 |
sass |
0.4.8 |
scales |
1.3.0 |
selectr |
0.4-2 |
sessioninfo |
1.2.2 |
shape |
1.4.6 |
shiny |
1.8.0 |
sourcetools |
0.1.7-1 |
sparklyr |
1.8.4 |
spatial |
7.3-15 |
splines |
4.3.2 |
sqldf |
0.4-11 |
SQUAREM |
2021.1 |
stats |
4.3.2 |
stats4 |
4.3.2 |
stringi |
1.8.3 |
stringr |
1.5.1 |
survival |
3.5-5 |
swagger |
3.33.1 |
sys |
3.4.2 |
systemfonts |
1.0.5 |
tcltk |
4.3.2 |
testthat |
3.2.1 |
textshaping |
0.3.7 |
tibble |
3.2.1 |
tidyr |
1.3.1 |
tidyselect |
1.2.0 |
tidyverse |
2.0.0 |
timechange |
0.3.0 |
timeDate |
4032.109 |
tinytex |
0.49 |
tools |
4.3.2 |
tzdb |
0.4.0 |
urlchecker |
1.0.1 |
usethis |
2.2.2 |
utf8 |
1.2.4 |
utils |
4.3.2 |
uuid |
1.2-0 |
V8 |
4.4.1 |
vctrs |
0.6.5 |
viridisLite |
0.4.2 |
vroom |
1.6.5 |
waldo |
0.5.2 |
whisker |
0.4.1 |
withr |
3.0.0 |
xfun |
0.41 |
xml2 |
1.3.6 |
xopen |
1.0.0 |
xtable |
1.8-4 |
yaml |
2.3.8 |
zeallot |
0.1.0 |
zip |
2.3.1 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.390 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.390 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.390 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.390 |
com.amazonaws |
aws-java-sdk-config |
1.12.390 |
com.amazonaws |
aws-java-sdk-core |
1.12.390 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.390 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.390 |
com.amazonaws |
aws-java-sdk-directory |
1.12.390 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.390 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.390 |
com.amazonaws |
aws-java-sdk-efs |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.390 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.390 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.390 |
com.amazonaws |
aws-java-sdk-emr |
1.12.390 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.390 |
com.amazonaws |
aws-java-sdk-glue |
1.12.390 |
com.amazonaws |
aws-java-sdk-iam |
1.12.390 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.390 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.390 |
com.amazonaws |
aws-java-sdk-kms |
1.12.390 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.390 |
com.amazonaws |
aws-java-sdk-logs |
1.12.390 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.390 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.390 |
com.amazonaws |
aws-java-sdk-rds |
1.12.390 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.390 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.390 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.390 |
com.amazonaws |
aws-java-sdk-ses |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.390 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.390 |
com.amazonaws |
aws-java-sdk-sns |
1.12.390 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.390 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.390 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.390 |
com.amazonaws |
aws-java-sdk-sts |
1.12.390 |
com.amazonaws |
aws-java-sdk-support |
1.12.390 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.390 |
com.amazonaws |
jmespath-java |
1.12.390 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
databricks-sdk-java |
0.17.1 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
compilerplugin_2.12 |
0.4.15-10 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.15.2 |
com.fasterxml.jackson.core |
jackson-core |
2.15.2 |
com.fasterxml.jackson.core |
jackson-databind |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-yaml |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-jsr310 |
2.16.0 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.15.2 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.15.2 |
com.github.ben-manes.caffeine |
caffeine |
2.9.3 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1-natives |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1-natives |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64 |
1.1-natives |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64 |
1.1-natives |
com.github.luben |
zstd-jni |
1.5.5-4 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.10.1 |
com.google.crypto.tink |
tink |
1.9.0 |
com.google.errorprone |
error_prone_annotations |
2.10.0 |
com.google.flatbuffers |
flatbuffers-java |
23.5.26 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
2.6.1 |
com.helger |
profiler |
1.1.1 |
com.jcraft |
jsch |
0.1.55 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.microsoft.sqlserver |
mssql-jdbc |
11.2.2.jre8 |
com.ning |
compress-lzf |
1.1.2 |
com.sun.mail |
javax.mail |
1.5.2 |
com.sun.xml.bind |
jaxb-core |
2.2.11 |
com.sun.xml.bind |
jaxb-impl |
2.2.11 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.4.3 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.uber |
h3 |
3.7.3 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.16.0 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.5 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.13.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
3.0.3 |
dev.ludovic.netlib |
blas |
3.0.3 |
dev.ludovic.netlib |
lapack |
3.0.3 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.25 |
io.delta |
delta-sharing-client_2.12 |
1.0.4 |
io.dropwizard.metrics |
metrics-annotation |
4.2.19 |
io.dropwizard.metrics |
metrics-core |
4.2.19 |
io.dropwizard.metrics |
metrics-graphite |
4.2.19 |
io.dropwizard.metrics |
metrics-healthchecks |
4.2.19 |
io.dropwizard.metrics |
metrics-jetty9 |
4.2.19 |
io.dropwizard.metrics |
metrics-jmx |
4.2.19 |
io.dropwizard.metrics |
metrics-json |
4.2.19 |
io.dropwizard.metrics |
metrics-jvm |
4.2.19 |
io.dropwizard.metrics |
metrics-servlets |
4.2.19 |
io.netty |
netty-all |
4.1.96.Final |
io.netty |
netty-buffer |
4.1.96.Final |
io.netty |
netty-codec |
4.1.96.Final |
io.netty |
netty-codec-http |
4.1.96.Final |
io.netty |
netty-codec-http2 |
4.1.96.Final |
io.netty |
netty-codec-socks |
4.1.96.Final |
io.netty |
netty-common |
4.1.96.Final |
io.netty |
netty-handler |
4.1.96.Final |
io.netty |
netty-handler-proxy |
4.1.96.Final |
io.netty |
netty-resolver |
4.1.96.Final |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-windows-x86_64 |
io.netty |
netty-tcnative-classes |
2.0.61.Final |
io.netty |
netty-transport |
4.1.96.Final |
io.netty |
netty-transport-classes-epoll |
4.1.96.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.96.Final |
io.netty |
netty-transport-native-epoll |
4.1.96.Final |
io.netty |
netty-transport-native-epoll |
4.1.96.Final-linux-aarch_64 |
io.netty |
netty-transport-native-epoll |
4.1.96.Final-linux-x86_64 |
io.netty |
netty-transport-native-kqueue |
4.1.96.Final-osx-aarch_64 |
io.netty |
netty-transport-native-kqueue |
4.1.96.Final-osx-x86_64 |
io.netty |
netty-transport-native-unix-common |
4.1.96.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.12.1 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.3 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.9.3 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.10.11 |
org.apache.ant |
ant-jsch |
1.10.11 |
org.apache.ant |
ant-launcher |
1.10.11 |
org.apache.arrow |
arrow-format |
15.0.0 |
org.apache.arrow |
arrow-memory-core |
15.0.0 |
org.apache.arrow |
arrow-memory-netty |
15.0.0 |
org.apache.arrow |
arrow-vector |
15.0.0 |
org.apache.avro |
avro |
1.11.3 |
org.apache.avro |
avro-ipc |
1.11.3 |
org.apache.avro |
avro-mapred |
1.11.3 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.23.0 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.10.0 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.datasketches |
datasketches-java |
3.1.0 |
org.apache.datasketches |
datasketches-memory |
2.0.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-runtime |
3.3.6 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.8.1 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.14 |
org.apache.httpcomponents |
httpcore |
4.4.16 |
org.apache.ivy |
ivy |
2.5.1 |
org.apache.logging.log4j |
log4j-1.2-api |
2.22.1 |
org.apache.logging.log4j |
log4j-api |
2.22.1 |
org.apache.logging.log4j |
log4j-core |
2.22.1 |
org.apache.logging.log4j |
log4j-slf4j2-impl |
2.22.1 |
org.apache.orc |
orc-core |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-mapreduce |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-shims |
1.9.2 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.ws.xmlschema |
xmlschema-core |
2.3.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.23 |
org.apache.yetus |
audience-annotations |
0.13.0 |
org.apache.zookeeper |
zookeeper |
3.6.3 |
org.apache.zookeeper |
zookeeper-jute |
3.6.3 |
org.checkerframework |
checker-qual |
3.31.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.collections |
eclipse-collections |
11.1.0 |
org.eclipse.collections |
eclipse-collections-api |
11.1.0 |
org.eclipse.jetty |
jetty-client |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-continuation |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-http |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-io |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-jndi |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-plus |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-proxy |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-security |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-server |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlet |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlets |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-webapp |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-xml |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.52.v20230823 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.40 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.40 |
org.glassfish.jersey.core |
jersey-client |
2.40 |
org.glassfish.jersey.core |
jersey-common |
2.40 |
org.glassfish.jersey.core |
jersey-server |
2.40 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.40 |
org.hibernate.validator |
hibernate-validator |
6.1.7.Final |
org.ini4j |
ini4j |
0.5.4 |
org.javassist |
javassist |
3.29.2-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mlflow |
mlflow-spark_2.12 |
2.9.1 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.6.1 |
org.roaringbitmap |
RoaringBitmap |
0.9.45 |
org.roaringbitmap |
shims |
0.9.45 |
org.rocksdb |
rocksdbjni |
8.3.2 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.15 |
org.scala-lang |
scala-library_2.12 |
2.12.15 |
org.scala-lang |
scala-reflect_2.12 |
2.12.15 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.11.0 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.2.15 |
org.scalanlp |
breeze-macros_2.12 |
2.1.0 |
org.scalanlp |
breeze_2.12 |
2.1.0 |
org.scalatest |
scalatest-compatible |
3.2.15 |
org.scalatest |
scalatest-core_2.12 |
3.2.15 |
org.scalatest |
scalatest-diagrams_2.12 |
3.2.15 |
org.scalatest |
scalatest-featurespec_2.12 |
3.2.15 |
org.scalatest |
scalatest-flatspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-freespec_2.12 |
3.2.15 |
org.scalatest |
scalatest-funspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-funsuite_2.12 |
3.2.15 |
org.scalatest |
scalatest-matchers-core_2.12 |
3.2.15 |
org.scalatest |
scalatest-mustmatchers_2.12 |
3.2.15 |
org.scalatest |
scalatest-propspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-refspec_2.12 |
3.2.15 |
org.scalatest |
scalatest-shouldmatchers_2.12 |
3.2.15 |
org.scalatest |
scalatest-wordspec_2.12 |
3.2.15 |
org.scalatest |
scalatest_2.12 |
3.2.15 |
org.slf4j |
jcl-over-slf4j |
2.0.7 |
org.slf4j |
jul-to-slf4j |
2.0.7 |
org.slf4j |
slf4j-api |
2.0.7 |
org.slf4j |
slf4j-simple |
1.7.25 |
org.threeten |
threeten-extra |
1.7.1 |
org.tukaani |
xz |
1.9 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.1.3.Final |
org.xerial |
sqlite-jdbc |
3.42.0.0 |
org.xerial.snappy |
snappy-java |
1.1.10.3 |
org.yaml |
snakeyaml |
2.0 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.cryptools |
AmazonCorrettoCryptoProvider |
1.6.1-linux-x86_64 |
software.amazon.ion |
ion-java |
1.0.2 |
stax |
stax-api |
1.0.1 |