Amazon
Web Services
Microsoft Azure
Google Cloud PlatformDatabricks Runtime 16.0 (Beta)
Beta
Databricks Runtime 16.0 is in Beta. The contents of the supported environments may change during the Beta. Changes can include the list of packages or versions of installed packages.
The following release notes provide information about Databricks Runtime 16.0, powered by Apache Spark 3.5.0.
Tip
To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.
Behavioral changes
Improve duplicate match detection in Delta Lake MERGE
In Databricks Runtime 15.4 LTS and below, MERGE operations fail if more than one row in the source table matches the same row in the target table based on the MERGE condition specified in the ON clause. In Databricks Runtime 16.0 and above, MERGE also considers conditions specified in the WHEN MATCHED clause. See Upsert into a Delta Lake table using merge.
SparkR is now deprecated
SparkR in Databricks is deprecated in Databricks Runtime 16.0 and above, in preparation for it being deprecated in the upcoming Spark 4 release. See the Apache Spark, Deprecate SparkR thread.
Databricks recommends using sparklyr instead.
Cluster library installation method can no longer be overridden
The Spark configurations spark.databricks.libraries.enableSparkPyPI, spark.databricks.libraries.enableMavenResolution, and spark.databricks.libraries.enableCRANResolutionnow now default to true and cannot be overridden.
JDK 17 is now the default
[Breaking change] On Databricks Runtime 16.0 and above, the default JDK version is switched from JDK 8 to JDK 17. This change is made because of the planned deprecation and end-of-support for JDK 8. This affects the following:
Java code running on Databricks compute must be compatible with Java 17.
Scala code running in notebooks or Databricks compute must be compatible with Java 17.
Java and Scala libraries installed on compute must be compatible with Java 17.
Versions of the Apache Hive metastore client below 2.x. Setting the Spark configuration
spark.sql.hive.metastore.versionto a version lower than 2.x will cause compatibility issues with Java 17 and connection failures to the Hive metastore. Databricks recommends upgrading Hive to a version above 2.0.0.
If you need to revert to Java 8, add the following to the Spark environment variables when you configure your Databricks compute:
JNAME=zulu8-ca-amd64
If you’re using ARM instances, use the following:
JNAME=zulu8-ca-arm64
To learn more about specifying JDK versions with Databricks compute, see Create a cluster that uses JDK 17.
For help with migrating your code from Java 8, see the following guides:
Default timeout of two hours for cluster-scoped library installs
On Databricks Runtime 16.0 and above, cluster-scoped library installation has a default timeout of two hours. Library installations that take longer than this timeout will fail and the installation is terminated. You can change the timeout period using the Spark configuration spark.databricks.driver.clusterLibraryInstallationTimeoutSec when you configure a cluster.
Installing libraries from DBFS and setting the spark conf spark.databricks.driver.dbfsLibraryInstallationAllowed are disabled
To improve the security of libraries in a Databricks workspace, installing libraries from DBFS is fully disabled on Databricks Runtime 16.0 and above. Additionally, you can no longer use the Spark configuration spark.databricks.driver.dbfsLibraryInstallationAllowed on Databricks Runtime 16.0 and above.
New features and improvements
Avro support for recursive schema
You can now use the option recursiveFieldMaxDepth with the from_avro function and the avro data source. This option sets the max depth for schema recursion on the Avro data source. See Read and write streaming Avro data.
Expanded support for Confluent Schema Registry for Avro
Databricks now supports Avro schema reference with the Confluent Schema Registry. See Authenticate to an external Confluent Schema Registry.
Force reclustering on tables with liquid clustering
In Databricks Runtime 16.0 and above, you can force reclustering of all records in a table with liquid clustering enabled with the syntax OPTIMIZE FULL. See Force reclustering for all records.
Improvements to %autoreload keep Python modules synced with source code
Starting with Databricks Runtime 16.0, when you import a Python module from workspace files, Databricks automatically suggests using %autoreload if the module has changed since its last import. See Autoreload for Python modules.
Python and Scala Delta API support for identity columns
You can now use Delta APIs for Python and Scala to create tables with identity columns. See Use identity columns in Delta Lake.
Create liquid clustered tables during streaming writes
You can now use clusterBy to enable liquid clustering while creating new tables with Structured Streaming writes. See Enable liquid clustering.
OPTIMIZE FULL
Databricks Runtime 16.0 supports OPTIMIZE FULL command. This clause optimizes the whole table including data that may have previously been clustered.
New SQL functions
The following new SQL functions have been added in Databricks Runtime 16.0:
-
This function provides an error tolerant version of url_decode that returns
NULLif the input is not a valid URL-encoded string. -
Returns 0 if the input is
NULL, or its input if it is notNULL. -
Returns
NULLif the input is 0, or its input if it is not 0.
Library upgrades
Upgraded Python libraries:
azure-core from 1.30.2 to 1.31.0
azure-storage-blob from 12.19.1 to 12.23.0
azure-storage-file-datalake from 12.14.0 to 12.17.0
black from 23.3.0 to 24.4.2
blinker from 1.4 to 1.7.0
boto3 from 1.34.39 to 1.34.69
botocore from 1.34.39 to 1.34.69
certifi from 2023.7.22 to 2024.6.2
cffi from 1.15.1 to 1.16.0
click from 8.0.4 to 8.1.7
comm from 0.1.2 to 0.2.1
contourpy from 1.0.5 to 1.2.0
cryptography from 41.0.3 to 42.0.5
Cython from 0.29.32 to 3.0.11
databricks-sdk from 0.20.0 to 0.30.0
dbus-python from 1.2.18 to 1.3.2
filelock from 3.13.4 to 3.15.4
fonttools from 4.25.0 to 4.51.0
GitPython from 3.1.43 to 3.1.37
google-api-core from 2.18.0 to 2.20.0
google-auth from 2.31.0 to 2.35.0
google-cloud-storage from 2.17.0 to 2.18.2
google-crc32c from 1.5.0 to 1.6.0
google-resumable-media from 2.7.1 to 2.7.2
googleapis-common-protos from 1.63.2 to 1.65.0
httplib2 from 0.20.2 to 0.20.4
idna from 3.4 to 3.7
ipykernel from 6.25.1 to 6.28.0
ipython from 8.15.0 to 8.25.0
jedi from 0.18.1 to 0.19.1
jmespath from 0.10.0 to 1.0.1
joblib from 1.2.0 to 1.4.2
jupyter_client from 7.4.9 to 8.6.0
jupyter_core from 5.3.0 to 5.7.2
launchpadlib from 1.10.16 to 1.11.0
lazr.restfulclient from 0.14.4 to 0.14.6
matplotlib from 3.7.2 to 3.8.4
mlflow-skinny from 2.11.4 to 2.15.1
more-itertools from 8.10.0 to 10.3.0
mypy-extensions from 0.4.3 to 1.0.0
nest-asyncio from 1.5.6 to 1.6.0
numpy from 1.23.5 to 1.26.4
oauthlib from 3.2.0 to 3.2.2
packaging from 23.2 to 24.1
patsy from 0.5.3 to 0.5.6
pip from 23.2.1 to 24.2
plotly from 5.9.0 to 5.22.0
prompt-toolkit from 3.0.36 to 3.0.43
pyarrow from 14.0.1 to 15.0.2
pydantic from 1.10.6 to 2.8.2
PyGObject from 3.42.1 to 3.48.2
PyJWT from 2.3.0 to 2.7.0
pyodbc from 4.0.38 to 5.0.1
python-dateutil from 2.8.2 to 2.9.0.post0
python-lsp-jsonrpc from 1.1.1 to 1.1.2
pytz from 2022.7 to 2024.1
PyYAML from 6.0 to 6.0.1
pyzmq from 23.2.0 to 25.1.2
requests from 2.31.0 to 2.32.2
scikit-learn from 1.3.0 to 1.4.2
scipy from 1.11.1 to 1.13.1
seaborn from 0.12.2 to 0.13.2
setuptools from 68.0.0 to 74.0.0
smmap from 5.0.1 to 5.0.0
sqlparse from 0.5.0 to 0.5.1
statsmodels from 0.14.0 to 0.14.2
tornado from 6.3.2 to 6.4.1
traitlets from 5.7.1 to 5.14.3
typing_extensions from 4.10.0 to 4.11.0
ujson from 5.4.0 to 5.10.0
virtualenv from 20.24.2 to 20.26.2
wheel from 0.38.4 to 0.43.0
zipp from 3.11.0 to 3.17.0
Upgraded R libraries:
arrow from 14.0.0.2 to 16.1.0
backports from 1.4.1 to 1.5.0
base from 4.3.2 to 4.4.0
bitops from 1.0-7 to 1.0-8
boot from 1.3-28 to 1.3-30
brio from 1.1.4 to 1.1.5
broom from 1.0.5 to 1.0.6
bslib from 0.6.1 to 0.8.0
cachem from 1.0.8 to 1.1.0
callr from 3.7.3 to 3.7.6
cli from 3.6.2 to 3.6.3
clock from 0.7.0 to 0.7.1
cluster from 2.1.4 to 2.1.6
codetools from 0.2-19 to 0.2-20
colorspace from 2.1-0 to 2.1-1
compiler from 4.3.2 to 4.4.0
crayon from 1.5.2 to 1.5.3
curl from 5.2.0 to 5.2.1
data.table from 1.15.0 to 1.15.4
datasets from 4.3.2 to 4.4.0
DBI from 1.2.1 to 1.2.3
dbplyr from 2.4.0 to 2.5.0
digest from 0.6.34 to 0.6.36
downlit from 0.4.3 to 0.4.4
evaluate from 0.23 to 0.24.0
farver from 2.1.1 to 2.1.2
fastmap from 1.1.1 to 1.2.0
foreign from 0.8-85 to 0.8-86
fs from 1.6.3 to 1.6.4
future from 1.33.1 to 1.34.0
future.apply from 1.11.1 to 1.11.2
gert from 2.0.1 to 2.1.0
ggplot2 from 3.4.4 to 3.5.1
gh from 1.4.0 to 1.4.1
globals from 0.16.2 to 0.16.3
graphics from 4.3.2 to 4.4.0
grDevices from 4.3.2 to 4.4.0
grid from 4.3.2 to 4.4.0
gt from 0.10.1 to 0.11.0
gtable from 0.3.4 to 0.3.5
hardhat from 1.3.1 to 1.4.0
highr from 0.10 to 0.11
htmltools from 0.5.7 to 0.5.8.1
httpuv from 1.6.14 to 1.6.15
httr2 from 1.0.0 to 1.0.2
ipred from 0.9-14 to 0.9-15
KernSmooth from 2.23-21 to 2.23-22
knitr from 1.45 to 1.48
lattice from 0.21-8 to 0.22-5
lava from 1.7.3 to 1.8.0
markdown from 1.12 to 1.13
MASS from 7.3-60 to 7.3-60.0.1
Matrix from 1.5-4.1 to 1.6-5
methods from 4.3.2 to 4.4.0
mgcv from 1.8-42 to 1.9-1
mlflow from 2.10.0 to 2.14.1
munsell from 0.5.0 to 0.5.1
nlme from 3.1-163 to 3.1-165
openssl from 2.1.1 to 2.2.0
parallel from 4.3.2 to 4.4.0
parallelly from 1.36.0 to 1.38.0
pkgbuild from 1.4.3 to 1.4.4
pkgdown from 2.0.7 to 2.1.0
pkgload from 1.3.4 to 1.4.0
processx from 3.8.3 to 3.8.4
prodlim from 2023.08.28 to 2024.06.25
promises from 1.2.1 to 1.3.0
ps from 1.7.6 to 1.7.7
ragg from 1.2.7 to 1.3.2
Rcpp from 1.0.12 to 1.0.13
RcppEigen from 0.3.3.9.4 to 0.3.4.0.0
reactR from 0.5.0 to 0.6.0
recipes from 1.0.9 to 1.1.0
remotes from 2.4.2.1 to 2.5.0
reprex from 2.1.0 to 2.1.1
rlang from 1.1.3 to 1.1.4
rmarkdown from 2.25 to 2.27
roxygen2 from 7.3.1 to 7.3.2
rpart from 4.1.21 to 4.1.23
RSQLite from 2.3.5 to 2.3.7
rstudioapi from 0.15.0 to 0.16.0
rvest from 1.0.3 to 1.0.4
sass from 0.4.8 to 0.4.9
shape from 1.4.6 to 1.4.6.1
shiny from 1.8.0 to 1.9.1
sparklyr from 1.8.4 to 1.8.6
spatial from 7.3-15 to 7.3-17
splines from 4.3.2 to 4.4.0
stats from 4.3.2 to 4.4.0
stats4 from 4.3.2 to 4.4.0
stringi from 1.8.3 to 1.8.4
survival from 3.5-5 to 3.6-4
swagger from 3.33.1 to 5.17.14.1
systemfonts from 1.0.5 to 1.1.0
tcltk from 4.3.2 to 4.4.0
testthat from 3.2.1 to 3.2.1.1
textshaping from 0.3.7 to 0.4.0
tidyselect from 1.2.0 to 1.2.1
tinytex from 0.49 to 0.52
tools from 4.3.2 to 4.4.0
usethis from 2.2.2 to 3.0.0
utils from 4.3.2 to 4.4.0
uuid from 1.2-0 to 1.2-1
V8 from 4.4.1 to 4.4.2
withr from 3.0.0 to 3.0.1
xfun from 0.41 to 0.46
xopen from 1.0.0 to 1.0.1
yaml from 2.3.8 to 2.3.10
Upgraded Java libraries:
com.google.protobuf.protobuf-java from 2.6.1 to 3.25.1
io.airlift.aircompressor from 0.25 to 0.27
io.delta.delta-sharing-client_2.12 from 1.1.2 to 1.2.0
org.rocksdb.rocksdbjni from 8.11.4 to 9.2.1
org.scalactic.scalactic_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-compatible from 3.2.15 to 3.2.16
org.scalatest.scalatest-core_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-diagrams_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-featurespec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-flatspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-freespec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-funspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-funsuite_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-matchers-core_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-mustmatchers_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-propspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-refspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-shouldmatchers_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest-wordspec_2.12 from 3.2.15 to 3.2.16
org.scalatest.scalatest_2.12 from 3.2.15 to 3.2.16
Apache Spark
Databricks Runtime 16.0 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.4 LTS, as well as the following additional bug fixes and improvements made to Spark:
[SPARK-49743] [ES-1260022][BEHAVE-157][SC-177475][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
[SPARK-49816] [SC-177896][SQL] Should only update out-going-ref-count for referenced outer CTE relation
[SPARK-48939] [SC-177022][SC-172766][AVRO] Support reading Avro with recursive schema reference
[SPARK-49688] [SC-177468][ES-1242349][CONNECT] Fix a data race between interrupt and execute plan
[SPARK-49771] [SC-177466][PYTHON] Improve Pandas Scalar Iter UDF error when output rows exceed input rows
[SPARK-48866] [SC-170772][SQL] Fix hints of valid charset in the error message of INVALID_PARAMETER_VALUE.CHARSET
[SPARK-48195] [FIXFORWARD][SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
[SPARK-49585] [CONNECT] Replace executions map in SessionHolder with operationID set
[SPARK-49211] [SC-174257][SQL] V2 Catalog can also support built-in data sources
[SPARK-49684] Minimise the lifetime of the session restoration lock
[SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPARK-48291] Structured log framework on the java side
[SPARK-48857] [SC-170661][SQL] Restrict charsets in CSVOptions
[SPARK-49152] [SC-173690][SQL] V2SessionCatalog should use V2Command
[SPARK-42846] [SC-176588][SQL] Remove error condition LEGACYERROR_TEMP_2011
[SPARK-48195] [SC-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
[SPARK-49630] [SC-177379][SS] Add flatten option to process collection types with state data source reader
[SPARK-49699] [SC-177154][SS] Disable PruneFilters for streaming workloads
[SPARK-48781] [SC-175282][SQL] Add Catalog APIs for loading stored procedures
[SPARK-49667] [SC-177068][SQL] Disallow CS_AI collators with expressions that use StringSearch
[SPARK-49737] [SC-177207][SQL] Disable bucketing on collated columns in complex types
[SPARK-48712] [SC-169794][SQL] Perf Improvement for encode with empty values or UTF-8 charset
[SPARK-49038] [SC-173933][SQL] SQLMetric should report the raw value in the accumulator update event
[SPARK-48541] [SC-169066][CORE] Add a new exit code for executors killed by TaskReaper
[SPARK-48774] [SC-170138][SQL] Use SparkSession in SQLImplicits
[SPARK-49719] [SC-177139][SQL] Make
UUIDandSHUFFLEaccept integerseed[SPARK-49713] [SC-177135][PYTHON][CONNECT] Make function
count_min_sketchaccept number arguments[SPARK-47601] [SC-162499][GRAPHX] Graphx: Migrate logs with variables to structured logging framework
[SPARK-49460] [SC-177237][SQL] Followup: fix potential NPE risk
[SPARK-49738] [SC-177219][SQL] Endswith bug fix
[SPARK-48623] [SC-170822][CORE] Structured logging migrations [Part 3]
[SPARK-49677] [SC-177148][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset
[SPARK-49684] [SC-177040][CONNECT] Remove global locks from session and execution managers
[SPARK-48302] [SC-168814][PYTHON] Preserve nulls in map columns in PyArrow Tables
[SPARK-48601] [SC-169025][SQL] Give a more user friendly error message when setting a null value for JDBC Option
[SPARK-48635] [SC-169263][SQL] Assign classes to join type errors and as-of join error
[SPARK-49673] [SC-177032][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
[SPARK-49693] [SC-177071][PYTHON][CONNECT] Refine the string representation of
timedelta[SPARK-49687] [SC-176901][SQL] Delay sorting in
validateAndMaybeEvolveStateSchema[SPARK-49718] [SC-177112][PS] Switch
Scatterplot to sampled data[SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings
[SPARK-48484] [SC-167484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts
[SPARK-48341] [SC-166560][CONNECT] Allow plugins to use QueryTest in their tests
[SPARK-42252] [SC-168723][CORE] Add
spark.shuffle.localDisk.file.output.bufferand deprecatespark.shuffle.unsafe.file.output.buffer[SPARK-48314] [SC-166565][SS] Don’t double cache files for FileStreamSource using Trigger.AvailableNow
[SPARK-49567] [SC-176241][PYTHON] Use
classicinstead ofvanillafrom PySpark code base[SPARK-48374] [SC-167596][PYTHON] Support additional PyArrow Table column types
[SPARK-48300] [SC-166481][SQL] Codegen Support for
from_xml[SPARK-49412] [SC-177059][PS] Compute all box plot metrics in single job
[SPARK-49692] [SC-177031][PYTHON][CONNECT] Refine the string representation of literal date and datetime
[SPARK-49392] [ES-1130351][SC-176705][SQL] Catch errors when failing to write to external data source
[SPARK-48306] [SC-166241][SQL] Improve UDT in error message
[SPARK-44924] [SC-166379][SS] Add config for FileStreamSource cached files
[SPARK-48176] [SC-165644][SQL] Adjust name of FIELD_ALREADY_EXISTS error condition
[SPARK-49691] [SC-176988][PYTHON][CONNECT] Function
substringshould accept column names[SPARK-49502] [SC-176077][CORE] Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
[SPARK-49244] [SC-176703][SQL] Further exception improvements for parser/interpreter
[SPARK-48355] [SC-176684][SQL] Support for CASE statement
[SPARK-49355] [SC-175121][SQL]
levenshteinshould check whether thecollationvalues of all parameter types are the same[SPARK-49640] [SC-176953][PS] Apply reservoir sampling in
SampledPlotBase[SPARK-49678] [SC-176857][CORE] Support
spark.test.masterinSparkSubmitArguments[SPARK-49680] [SC-176856][PYTHON] Limit
Sphinxbuild parallelism to 4 by default[SPARK-49396] Revert “[SC-176030][SQL] Modify nullability check for CaseWhen expression”
[SPARK-48419] [SC-167443][SQL] Foldable propagation replace foldable column shoul…
[SPARK-49556] [SC-176757][SQL] Add SQL pipe syntax for the SELECT operator
[SPARK-49438] [SC-175237][SQL] Fix the pretty name of the
FromAvro&ToAvroexpression[SPARK-49659] [SC-1229924][SQL] Add a nice user-facing error for scalar subqueries inside VALUES clause
[SPARK-49646] [SC-176778][SQL] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
[SPARK-49354] [SC-175034][SQL]
split_partshould check whether thecollationvalues of all parameter types are the same[SPARK-49478] [SC-175914][CONNECT] Handle null metrics in ConnectProgressExecutionListener
[SPARK-48358] [SC-176374][SQL] Support for REPEAT statement
[SPARK-49183] [SC-173680][SQL] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION
[SPARK-49611] [SC-176791][SQL] Introduce TVF
collations()& remove theSHOW COLLATIONScommand[SPARK-49261] [SC-176589][SQL] Don’t replace literals in aggregate expressions with group-by expressions
[SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
[SPARK-49594] [SC-176569][SS] Adding check on whether columnFamilies were added or removed to write StateSchemaV3 file
[SPARK-49578] [SC-176385][SQL] Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
[SPARK-48882] [SC-174256][SS] Assign names to streaming output mode related error classes
[SPARK-49155] [SC-176506][SQL][SS] Use more appropriate parameter type to construct
GenericArrayData[SPARK-49519] [SC-176388][SQL] Merge options of table and relation when constructing FileScanBuilder
[SPARK-49591] [SC-176587][SQL] Add Logical Type column to variant readme
[SPARK-49596] [SC-176423][SQL] Improve performance of
FormatString[SPARK-49525] [SC-176044][SS][CONNECT] Minor log improvement to Server Side Streaming Query ListenerBus Listener
[SPARK-49583] [SC-176272][SQL] Define the error sub-condition
SECONDS_FRACTIONfor invalid seconds fraction pattern[SPARK-49536] [SC-176242] Handle error in python streaming data source record prefetching
[SPARK-49443] [SC-176273][SQL][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
[SPARK-49544] [SASP-3990][SC-176557][CONNECT] Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
[SPARK-49548] [SASP-3990][SC-176556][CONNECT] Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
[SPARK-49551] [SC-176218][SS] Improve RocksDB log for replayChangelog
[SPARK-49595] [SC-176396][CONNECT][SQL] Fix
DataFrame.unpivot/meltin Spark Connect Scala Client[SPARK-49006] [SC-176162] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files
[SPARK-49600] [SC-176426][PYTHON] Remove
Python 3.6 and older-related logic fromtry_simplify_traceback[SPARK-49303] [SC-176013][SS] Implement TTL for ValueState in transformWithStateInPandas API
[SPARK-49191] [SC-176243][SS] Add support for reading transformWithState map state variables with state data source reader
[SPARK-49593] [SC-176371][SS] Throw RocksDB exception to the caller on DB close if an error is seen
[SPARK-49334] [SC-174803][SQL]
str_to_mapshould check whether thecollationvalues of all parameter types are the same[SPARK-42204] [SC-176126][CORE] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs
[SPARK-49575] [SC-176256][SS] Add logging for lock release only if acquiredThreadInfo is not null
[SPARK-49539] [SC-176250][SS] Update internal col families start identifier to a different one
[SPARK-49205] [SC-173853][SQL] KeyGroupedPartitioning should inherit HashPartitioningLike
[SPARK-49396] [SC-176030][SQL] Modify nullability check for CaseWhen expression
[SPARK-49476] [SC-175700][SQL] Fix nullability of base64 function
[SPARK-47262] [SC-174829][SQL] Assign names to error conditions for parquet conversions
[SPARK-47247] [SC-158349][SQL] Use smaller target size when coalescing partitions with exploding joins
[SPARK-49501] [SC-176166][SQL] Fix double-escaping of table location
[SPARK-49083] [SC-173214][CONNECT] Allow from_xml and from_json to natively work with json schemas
[SPARK-49043] [SC-174673][SQL] Fix interpreted codepath group by on map containing collated strings
[SPARK-48986] [SC-172265][CONNECT][SQL] Add ColumnNode Intermediate Representation
[SPARK-49326] [SC-176016][SS] Classify Error class for Foreach sink user function error
[SPARK-48348] [SC-175950][SPARK-48376][SQL] Introduce
LEAVEandITERATEstatements[SPARK-49523] [SC-175949][CONNECT] Increase maximum wait time for connect server to come up for testing
[SPARK-49000] [BEHAVE-105][ES-1194747][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates - DBR version 16.x
[SPARK-49311] [SC-175038][SQL] Make it possible for large ‘interval second’ values to be cast to decimal
[SPARK-49200] [SC-173699][SQL] Fix null type non-codegen ordering exception
[SPARK-49467] [SC-176051][SS] Add support for state data source reader and list state
[SPARK-47307] [SC-170891][SQL] Add a config to optionally chunk base64 strings
[SPARK-49391] [SC-176032][PS] Box plot select outliers by distance from fences
[SPARK-49445] [SC-175845][UI] Support show tooltip in the progress bar of UI
[SPARK-49451] [SC-175702] Allow duplicate keys in parse_json.
[SPARK-49275] [SC-175701][SQL] Fix return type nullness of the xpath expression
[SPARK-49021] [SC-175578][SS] Add support for reading transformWithState value state variables with state data source reader
[SPARK-49474] [BEHAVE-143][SC-169253][SC-175933][SS] Classify Error class for FlatMapGroupsWithState user function error
[SPARK-49408] [SC-175932][SQL] Use IndexedSeq in ProjectingInternalRow
[SPARK-49509] [SC-175853][CORE] Use
Platform.allocateDirectBufferinstead ofByteBuffer.allocateDirect[SPARK-49382] [SC-175013][PS] Make frame box plot properly render the fliers/outliers
[SPARK-49002] [SC-172846][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
[SPARK-49480] [SC-175699][CORE] Fix NullPointerException from
SparkThrowableHelper.isInternalError[SPARK-49477] [SC-175828][PYTHON] Improve pandas udf invalid return type error message
[SPARK-48693] [SC-169492][SQL] Simplify and unify toString of Invoke and StaticInvoke
[SPARK-49441] [SC-175716][ML]
StringIndexersort arrays in executors[SPARK-49347] [SC-175004][R] Deprecate SparkR
[SPARK-49357] [SC-175227][CONNECT][PYTHON] Vertically truncate deeply nested protobuf message
[SPARK-41982] [SC-120604][SQL] Partitions of type string should not be treated as numeric types
[SPARK-48776] [SC-170452][BEHAVE-72] Fix timestamp formatting for json, xml and csv
[SPARK-49223] [SC-174800][ML] Simplify the StringIndexer.countByValue with builtin functions
[SPARK-49016] Revert “[SC-174663][SQL] Restore the behavior thatqueries from raw CSV files are disallowed when only include corrupt record column and assign name to
_LEGACY_ERROR_TEMP_1285”[SPARK-49041] [SC-172392][PYTHON][CONNECT] Raise proper error for
dropDuplicateswhen wrongsubsetis given[SPARK-49050] [SC-175235] Enabling deleteIfExists operator in TWS with Virtual Column Families
[SPARK-49216] [SC-173919][CORE]Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
[SPARK-49252] [SC-175596][CORE] Make`TaskSetExcludeList` and
HeathTrackerindependent[SPARK-49352] [SC-174971][SQL] Avoid redundant array transform for identical expression
[SPARK-42307] [SC-173863][SQL] Assign name for error LEGACYERROR_TEMP_2232
[SPARK-49197] [SC-173732][CORE] Redact
Spark Commandoutput inlaunchermodule[SPARK-48913] [SC-173934][SQL] Implement IndentingXMLStreamWriter
[SPARK-49306] [SC-175363][PYTHON][SQL] Create SQL function aliases for ‘zeroifnull’ and ‘nullifzero’
[SPARK-48344] [SQL] SQL Scripting execution (including Spark Connect)
[SPARK-49402] [SC-175122][PYTHON] Fix Binder integration in PySpark documentation
[SPARK-49017] [SC-174664][SQL] Insert statement fails when multiple parameters are being used
[SPARK-49318] [SC-174733][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience
[SPARK-49016] [SC-174663][SQL] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to
_LEGACY_ERROR_TEMP_1285[SPARK-49387] [SC-175124][PYTHON] Fix type hint for
accuracyinpercentile_approxandapprox_percentile[SPARK-49131] [SC-174666][SS] TransformWithState should properly set implicit grouping keys even with lazy iterators
[SPARK-49301] [SC-174795][SS] Chunk arrow data passed to Python worker
[SPARK-49039] [SC-174651][UI] Reset checkbox when executor metrics are loaded in the Stages tab
[SPARK-48428] [SC-169806][SQL]: Fix IllegalStateException in NestedColumnAliasing
[SPARK-49353] [SC-174830][SQL] Update docs related to
UTF-32encoding/decoding[SPARK-48613] [SC-170966][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys
[SPARK-47473] [SC-160450][BEHAVE-127][SQL] Fix correctness issue of converting postgres INFINITY timestamps
[SPARK-49142] [SC-173658][CONNECT][PYTHON] Follow up to revert proto to string performance cost
[SPARK-49300] [SC-175008][CORE] Fix Hadoop delegation token leak when tokenRenewalInterval is not set.
[SPARK-49367] [SC-175012][PS] Parallelize the KDE computation for multiple columns (plotly backend)
[SPARK-49365] [SC-175011][PS] Simplify the bucket aggregation in hist plot
[SPARK-49372] [SC-175003][SS] Ensure that latestSnapshot is set to none on close to avoid subsequent use
[SPARK-49341] [SC-174785] Remove
connector/dockerin favor ofApache Spark Operator[SPARK-49344] [SC-174894][PS] Support
json_normalizefor Pandas API on Spark[SPARK-49306] [SC-174794][SQL] Create new SQL functions ‘zeroifnull’ and ‘nullifzero’
[SPARK-48796] [SC-174668][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
[SPARK-49342] [SC-174899][SQL] Make TO_AVRO SQL function ‘jsonFormatSchema’ argument optional
[SPARK-48628] [SC-174695][CORE] Add task peak on/off heap memory metrics
[SPARK-47407] [SC-159379][BEHAVE-126][SQL] Support java.sql.Types.NULL map to NullType
[SPARK-48628] [SC-173407][CORE] Add task peak on/off heap memory metrics
[SPARK-49166] [SC-173987][SQL] Support OFFSET in correlated subquery
[SPARK-49269] [SC-174676][SQL] Eagerly evaluate VALUES() list in AstBuilder
[SPARK-49281] [SC-174782][SQL] Optimze parquet binary getBytes with getBytesUnsafe to avoid copy cost
[SPARK-49113] [SC-174734] Dont assert on translation bugs - silently swallow the exception
[SPARK-49098] [SC-173253][SQL] Add write options for INSERT
[SPARK-48638] [SC-174694][FOLLOW][CONNECT] Fix documentation for ExecutionInfo
[SPARK-49250] [ES-1222826][SQL] Improve error message for nested UnresolvedWindowExpression in CheckAnalysis
[SPARK-48755] [SC-174258][SS][PYTHON] transformWithState pyspark base implementation and ValueState support
[SPARK-48966] [SC-174329][SQL] Improve error message with invalid unresolved column reference in UDTF call
[SPARK-46590] [SC-154115][SQL] Fix coalesce failed with unexpected partition indeces
[SPARK-49235] [SC-174159][SQL] Refactor ResolveInlineTables rule so it doesn’t traverse the whole tree
[SPARK-49060] [SC-173107][CONNECT] Clean up Mima rules for SQL-Connect binary compatibility checks
[SPARK-48762] [SC-172525][SQL] Introduce clusterBy DataFrameWriter API for Python
[SPARK-49207] [SC-173852][SQL] Fix one-to-many case mapping in SplitPart and StringSplitSQL
[SPARK-49204] [SC-173850][SQL] Fix surrogate pair handling in StringInstr and StringLocate
[SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
[SPARK-49204] [SC-173849][SQL] Fix surrogate pair handling in SubstringIndex
[SPARK-49204] [SC-173848][SQL] Fix surrogate pair handling in StringTrim
[SPARK-48967] [SC-173993]Fix forward SparkConfigOwnershipSuite test for OPTIMIZE_INSERT_INTO_VALUES_PARSER
[SPARK-49204] [SC-173851][SQL] Fix surrogate pair handling in StringReplace
[SPARK-48967] [SC-173993][SQL][16.x] Improve performance and memory footprint of “INSERT INTO … VALUES” Statements
[SPARK-49099] Revert “[SC-173229][SQL] CatalogManager.setCurrent…
[SPARK-48347] [SC-173812][SQL] Support for WHILE statement
[SPARK-49128] [SC-173344][CORE] Support custom History Server UI title
[SPARK-49146] [SC-173825][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
[SPARK-45787] [SC-172197][SQL] Support Catalog.listColumns for clustering columns
[SPARK-49099] [SC-173229][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
[SPARK-49138] [SC-173483][SQL] Fix CollationTypeCasts of several expressions
[SPARK-49163] [SC-173666][SQL] Attempt to create table based on broken parquet partition data results should return user-facing error
[SPARK-49201] [SC-173793][PS][PYTHON][CONNECT] Reimplement
histplot with Spark SQL[SPARK-49188] [SC-173682][SQL] Internal error on concat_ws called on array of arrays of string
[SPARK-49137] [SC-173677][SQL] When the Boolean condition in the
if statementis invalid, an exception should be thrown[SPARK-49193] [SC-173672][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
[SPARK-46632] [SC-173697][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
[SPARK-49078] [SC-173078][SQL] Support show columns syntax in v2 table
[SPARK-49141] [SC-173388][SQL] Mark variant as hive incompatible data type
[SPARK-49059] [Cherry-Pick][15.x][SC-172528][CONNECT] Move
SessionHolder.forTesting(...)to the test package[SPARK-49111] [SC-173661][SQL] Move withProjectAndFilter to the companion object of DataSourceV2Strategy
[SPARK-49185] [SC-173688][PS][PYTHON][CONNECT] Reimplement
kdeplot with Spark SQL[SPARK-49178] [SC-173673][SQL] Optimize performance of
Row#getSeqto match the performance when using Spark 3.5 with Scala 2.12[SPARK-49093] [SC-172958][SQL] GROUP BY with MapType nested inside complex type
[SPARK-49142] [SC-173469][CONNECT][PYTHON] Lower Spark Connect client log level to debug
[SPARK-48761] [SC-172048][SQL] Introduce clusterBy DataFrameWriter API for Scala
[SPARK-48346] [SC-173083][SQL] Support for IF ELSE statements in SQL scripts
[SPARK-48338] [SC-173112][SQL] Improve exceptions thrown from parser/interpreter
[SPARK-48658] [SC-169474][SQL] Encode/Decode functions report coding errors instead of mojibake for unmappable characters
[SPARK-49071] [SC-172954][SQL] Remove ArraySortLike trait
[SPARK-49107] Revert “Revert “[SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType””[SPARK-49070] [SC-172907][SS][SQL] TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
[SPARK-49114] [SC-173217] Sub-categorize cannot load state store errors
[SPARK-49107] Revert “[SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType”[SPARK-49048] [SC-173223][SS] Add support for reading relevant operator metadata at given batch id
[SPARK-49094] [SC-173049][SQL] Fix ignoreCorruptFiles non-functioning for hive orc impl with mergeSchema off
[SPARK-49108] [SC-173102][EXAMPLE] Add
submit_pi.shREST API example[SPARK-49107] [SC-173103][SQL]
ROUTINE_ALREADY_EXISTSsupports RoutineType[SPARK-48997] [SC-172484][SS] Implement individual unloads for maintenance thread pool thread failures
[SPARK-49063] [SC-173094][SQL] Fix Between with ScalarSubqueries
[SPARK-45891] [SC-172305][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec
[SPARK-49074] [BEHAVE-110][SC-172815][SQL] Fix variant with
df.cache()[SPARK-49003] [SC-172613][SQL] Fix interpreted code path hashing to be collation aware
[SPARK-48740] [SC-172430][SQL] Catch missing window specification error early
[SPARK-48999] [SC-172245][SS] Divide PythonStreamingDataSourceSimpleSuite
[SPARK-49031] [SC-172602] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2
[SPARK-49053] [SC-172494][PYTHON][ML] Make model save/load helper functions accept spark session
[SPARK-49032] [Backport][15.x][SS] Add schema path in metadata table entry, verify expected version and add operator metadata related test for operator metadata format v2
[SPARK-49034] [SC-172306][CORE] Support server-side
sparkPropertiesreplacement in REST Submission API[SPARK-48931] [SC-171895][SS] Reduce Cloud Store List API cost for state store maintenance task
[SPARK-48849] [SC-172068][SS]Create OperatorStateMetadataV2 for the TransformWithStateExec operator
[SPARK-49013] [SC-172322] Change key in collationsMap for Map and Array types in scala
[SPARK-48414] [SC-171884][PYTHON] Fix breaking change in python’s
fromJson[SPARK-48910] [SC-171001][SQL] Use HashSet/HashMap to avoid linear searches in PreprocessTableCreation
[SPARK-49007] [SC-172204][CORE] Improve
MasterPageto support custom title[SPARK-49009] [SC-172263][SQL][PYTHON] Make Column APIs and functions accept Enums
[SPARK-49033] [SC-172303][CORE] Support server-side
environmentVariablesreplacement in REST Submission API[SPARK-48363] [SC-166470][SQL] Cleanup some redundant codes in
from_xml[SPARK-46743] [SC-170867][SQL][BEHAVE-84] Count bug after ScalarSubqery is folded if it has an empty relation
[SPARK-49040] [SC-172351][SQL] Fix doc
sql-ref-syntax-aux-exec-imm.md[SPARK-48998] [SC-172212][ML] Meta algorithms save/load model with SparkSession
[SPARK-48959] [SC-171708][SQL] Make
NoSuchNamespaceExceptionextendNoSuchDatabaseExceptionto restore the exception handling[SPARK-48996] [SC-172130][SQL][PYTHON] Allow bare literals for and and or of Column
[SPARK-48990] [SC-171936] Follow up for #101759 - test fix
[SPARK-48338] [SC-171912][SQL] Check variable declarations
[SPARK-48990] [SC-171936][SQL] Unified variable related SQL syntax keywords
[SPARK-48988] [SC-171915][ML] Make
DefaultParamsReader/Writerhandle metadata with spark session[SPARK-48974] [SC-171978][SQL][SS][ML][MLLIB] Use
SparkSession.implicitsinstead ofSQLContext.implicits[SPARK-48760] [SC-170870][SQL] Fix CatalogV2Util.applyClusterByChanges
[SPARK-48928] [SC-171956] Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
[SPARK-48760] [SC-170139][SQL] Introduce ALTER TABLE … CLUSTER BY SQL syntax to change clustering columns
[SPARK-48844] Revert “[SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty”
[SPARK-48833] [SC-171914][SQL][VARIANT] Support variant in
InMemoryTableScan[SPARK-48975] [SC-171894][PROTOBUF] Remove unnecessary
ScalaReflectionLockdefinition fromprotobuf[SPARK-48970] [SC-171800][PYTHON][ML] Avoid using SparkSession.getActiveSession in spark ML reader/writer
[SPARK-48844] [SC-170669][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty
[SPARK-48714] [SC-170136] Fix failing df.mergeInto tests in PySpark and UC
[SPARK-48957] [SC-171797][SS] Return sub-classified error class on state store load for hdfs and rocksdb provider
[SPARK-48891] [Backport][15x][SC-171677][SS] Refactor StateSchemaCompatibilityChecker to unify all state schema formats
[SPARK-48972] [SC-171795][PYTHON] Unify the literal string handling in functions
[SPARK-48388] [SC-171337][SQL] Fix SET statement behavior for SQL Scripts
[SPARK-48743] [SC-170552][SQL][SS] MergingSessionIterator should better handle when getStruct returns null
[SPARK-48623] [15.x][SC-171322][CORE] Migrate FileAppender logs to structured logging
[SPARK-36680] [DBRRM-1123] Revert “[SC-170640][SQL] Supports Dynamic Table Options for Spark SQL”
[SPARK-48841] [SC-170868][BEHAVE-83][SQL] Include
collationNametosql()ofCollate[SPARK-48941] [SC-171703][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-48938] [SC-171577][PYTHON] Improve error messages when registering Python UDTFs
[SPARK-48350] [SC-171040][SQL] Introduction of Custom Exceptions for Sql Scripting
[SPARK-48907] [SC-171158][SQL] Fix the value
explicitTypesinCOLLATION_MISMATCH.EXPLICIT[SPARK-48945] [SC-171658][PYTHON] Simplify regex functions with
lit[SPARK-48944] [SC-171576][CONNECT] Unify the JSON-format schema handling in Connect Server
[SPARK-48836] [SC-171569] Integrate SQL schema with state schema/metadata
[SPARK-48946] [SC-171504][SQL] NPE in redact method when session is null
[SPARK-48921] [SC-171412][SQL] ScalaUDF encoders in subquery should be resolved for MergeInto
[SPARK-45155] [SC-171048][CONNECT] Add API Docs for Spark Connect JVM/Scala Client
[SPARK-48900] [SC-171319] Add
reasonfield forcancelJobGroupandcancelJobsWithTag[SPARK-48865] [SC-171154][SQL] Add try_url_decode function
[SPARK-48851] [SC-170767][SQL] Change the value of
SCHEMA_NOT_FOUNDfromnamespacetocatalog.namespace[SPARK-48510] [SC-170893][2/2] Support UDAF
toColumnAPI in Spark Connect[SPARK-45190] [SC-171055][SPARK-48897][PYTHON][CONNECT] Make
from_xmlsupport StructType schema[SPARK-48930] [SC-171304][CORE] Redact
awsAccessKeyIdby includingaccesskeypattern[SPARK-48909] [SC-171080][ML][MLLIB] Uses SparkSession over SparkContext when writing metadata
[SPARK-48883] [SC-171133][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-48924] [SC-171313][PS] Add a pandas-like
make_intervalhelper function[SPARK-48884] [SC-171051][PYTHON] Remove unused helper function
PythonSQLUtils.makeInterval[SPARK-48817] [SC-170636][SQL] Eagerly execute union multi commands together
[SPARK-48896] [SC-171079][ML][MLLIB] Avoid repartition when writing out the metadata
[SPARK-48892] [SC-171127][ML] Avoid per-row param read in
Tokenizer[SPARK-48927] [SC-171227][CORE] Show the number of cached RDDs in
StoragePage[SPARK-48886] [15.x][Backport][SC-171039][SS] Add version info to changelog v2 to allow for easier evolution
[SPARK-48903] [SC-171136][SS] Set the RocksDB last snapshot version correctly on remote load
[SPARK-48742] [SC-170538][SS] Virtual Column Family for RocksDB
[SPARK-48726] [15.x][SC-170753][SS] Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator
[SPARK-48794] [SC-170882][CONNECT][15.x] df.mergeInto support for Spark Connect (Scala and Python)
[SPARK-48714] [SC-170136][PYTHON] Implement
DataFrame.mergeIntoin PySpark[SPARK-48772] [SC-170642][SS][SQL] State Data Source Change Feed Reader Mode
[SPARK-48666] [SC-170887][SQL] Do not push down filter if it contains PythonUDFs
[SPARK-48845] [SC-170889][SQL] GenericUDF catch exceptions from children
[SPARK-48880] [SC-170974][CORE] Avoid throw NullPointerException if driver plugin fails to initialize
[SPARK-48888] [Backport][15x][SC-170973][SS] Remove snapshot creation based on changelog ops size
[SPARK-48871] [SC-170876] Fix INVALID_NON_DETERMINISTIC_EXPRESSIONS validation in…
[SPARK-48883] [SC-170894][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
[SPARK-36680] [SC-170640][SQL] Supports Dynamic Table Options for Spark SQL
[SPARK-48804] [SC-170558][SQL] Add classIsLoadable & OutputCommitter.isAssignableFrom check for output committer class configrations
[SPARK-46738] [SC-170791][PYTHON] Reenable a group of doctests
[SPARK-48858] [SC-170756][PYTHON] Remove deprecated
setDaemonmethod call ofThreadinlog_communication.py[SPARK-48639] [SC-169801][CONNECT][PYTHON] Add Origin to RelationCommon
[SPARK-48863] [SC-170770][ES-1133940][SQL] Fix ClassCastException when parsing JSON with “spark.sql.json.enablePartialResults” enabled
[SPARK-48343] [SC-170450][SQL] Introduction of SQL Scripting interpreter
[SPARK-48529] [SC-170755][SQL] Introduction of Labels in SQL Scripting
[SPARK-45292] Revert “[SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader”
[SPARK-48037] [SC-165330][CORE][3.5] Fix SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
[SPARK-48720] [SC-170551][SQL] Align the command
ALTER TABLE ... UNSET TBLPROPERTIES ...in v1 and v2[SPARK-48485] [SC-167825][CONNECT][SS] Support interruptTag and interruptAll in streaming queries
[SPARK-45292] [SC-151609][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader
[SPARK-48668] [SC-169815][SQL] Support ALTER NAMESPACE … UNSET PROPERTIES in v2
[SPARK-47914] [SC-165313][SQL] Do not display the splits parameter in Range
[SPARK-48807] [SC-170643][SQL] Binary Support for CSV datasource
[SPARK-48220] [SC-167592][PYTHON][15.X] Allow passing PyArrow Table to createDataFrame()
[SPARK-48545] [SC-169543][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents
[SPARK-47577] [SC-168875][SPARK-47579] Correct misleading usage of log key TASK_ID
Databricks ODBC/JDBC driver support
Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).
System environment
Operating System: Ubuntu 24.04.1 LTS
Java: Zulu17.50+19-CA
Scala: 2.12.18
Python: 3.12.3
R: 4.4.0
Delta Lake: 3.2.1
Installed Python libraries
Library |
Version |
Library |
Version |
Library |
Version |
|---|---|---|---|---|---|
annotated-types |
0.7.0 |
asttokens |
2.0.5 |
astunparse |
1.6.3 |
autocommand |
2.2.2 |
azure-core |
1.31.0 |
azure-storage-blob |
12.23.0 |
azure-storage-file-datalake |
12.17.0 |
backports.tarfile |
1.2.0 |
black |
24.4.2 |
blinker |
1.7.0 |
boto3 |
1.34.69 |
botocore |
1.34.69 |
cachetools |
5.3.3 |
certifi |
2024.6.2 |
cffi |
1.16.0 |
chardet |
4.0.0 |
charset-normalizer |
2.0.4 |
click |
8.1.7 |
cloudpickle |
2.2.1 |
comm |
0.2.1 |
contourpy |
1.2.0 |
cryptography |
42.0.5 |
cycler |
0.11.0 |
Cython |
3.0.11 |
databricks-sdk |
0.30.0 |
dbus-python |
1.3.2 |
debugpy |
1.6.7 |
decorator |
5.1.1 |
Deprecated |
1.2.14 |
distlib |
0.3.8 |
docstring-to-markdown |
0.11 |
entrypoints |
0.4 |
executing |
0.8.3 |
facets-overview |
1.1.1 |
filelock |
3.15.4 |
fonttools |
4.51.0 |
gitdb |
4.0.11 |
GitPython |
3.1.37 |
google-api-core |
2.20.0 |
google-auth |
2.35.0 |
google-cloud-core |
2.4.1 |
google-cloud-storage |
2.18.2 |
google-crc32c |
1.6.0 |
google-resumable-media |
2.7.2 |
googleapis-common-protos |
1.65.0 |
grpcio |
1.60.0 |
grpcio-status |
1.60.0 |
httplib2 |
0.20.4 |
idna |
3.7 |
importlib-metadata |
6.0.0 |
importlib_resources |
6.4.0 |
inflect |
7.3.1 |
ipyflow-core |
0.0.198 |
ipykernel |
6.28.0 |
ipython |
8.25.0 |
ipython-genutils |
0.2.0 |
ipywidgets |
7.7.2 |
isodate |
0.6.1 |
jaraco.context |
5.3.0 |
jaraco.functools |
4.0.1 |
jaraco.text |
3.12.1 |
jedi |
0.19.1 |
jmespath |
1.0.1 |
joblib |
1.4.2 |
jupyter_client |
8.6.0 |
jupyter_core |
5.7.2 |
kiwisolver |
1.4.4 |
launchpadlib |
1.11.0 |
lazr.restfulclient |
0.14.6 |
lazr.uri |
1.0.6 |
matplotlib |
3.8.4 |
matplotlib-inline |
0.1.6 |
mccabe |
0.7.0 |
mlflow-skinny |
2.15.1 |
more-itertools |
10.3.0 |
mypy |
1.10.0 |
mypy-extensions |
1.0.0 |
nest-asyncio |
1.6.0 |
nodeenv |
1.9.1 |
numpy |
1.26.4 |
oauthlib |
3.2.2 |
opentelemetry-api |
1.27.0 |
opentelemetry-sdk |
1.27.0 |
opentelemetry-semantic-conventions |
0.48b0 |
packaging |
24.1 |
pandas |
1.5.3 |
parso |
0.8.3 |
pathspec |
0.10.3 |
patsy |
0.5.6 |
pexpect |
4.8.0 |
pillow |
10.3.0 |
pip |
24.2 |
platformdirs |
3.10.0 |
plotly |
5.22.0 |
pluggy |
1.0.0 |
prompt-toolkit |
3.0.43 |
proto-plus |
1.24.0 |
protobuf |
4.24.1 |
psutil |
5.9.0 |
psycopg2 |
2.9.3 |
ptyprocess |
0.7.0 |
pure-eval |
0.2.2 |
pyarrow |
15.0.2 |
pyasn1 |
0.4.8 |
pyasn1-modules |
0.2.8 |
pyccolo |
0.0.52 |
pycparser |
2.21 |
pydantic |
2.8.2 |
pydantic_core |
2.20.1 |
pyflakes |
3.2.0 |
Pygments |
2.15.1 |
PyGObject |
3.48.2 |
PyJWT |
2.7.0 |
pyodbc |
5.0.1 |
pyparsing |
3.0.9 |
pyright |
1.1.294 |
python-dateutil |
2.9.0.post0 |
python-lsp-jsonrpc |
1.1.2 |
python-lsp-server |
1.10.0 |
pytoolconfig |
1.2.6 |
pytz |
2024.1 |
PyYAML |
6.0.1 |
pyzmq |
25.1.2 |
requests |
2.32.2 |
rope |
1.12.0 |
rsa |
4.9 |
s3transfer |
0.10.2 |
scikit-learn |
1.4.2 |
scipy |
1.13.1 |
seaborn |
0.13.2 |
setuptools |
74.0.0 |
six |
1.16.0 |
smmap |
5.0.0 |
sqlparse |
0.5.1 |
ssh-import-id |
5.11 |
stack-data |
0.2.0 |
statsmodels |
0.14.2 |
tenacity |
8.2.2 |
threadpoolctl |
2.2.0 |
tokenize-rt |
4.2.1 |
tomli |
2.0.1 |
tornado |
6.4.1 |
traitlets |
5.14.3 |
typeguard |
4.3.0 |
types-protobuf |
3.20.3 |
types-psutil |
5.9.0 |
types-pytz |
2023.3.1.1 |
types-PyYAML |
6.0.0 |
types-requests |
2.31.0.0 |
types-setuptools |
68.0.0.0 |
types-six |
1.16.0 |
types-urllib3 |
1.26.25.14 |
typing_extensions |
4.11.0 |
ujson |
5.10.0 |
unattended-upgrades |
0.1 |
urllib3 |
1.26.16 |
virtualenv |
20.26.2 |
wadllib |
1.3.6 |
wcwidth |
0.2.5 |
whatthepatch |
1.0.2 |
wheel |
0.43.0 |
wrapt |
1.14.1 |
yapf |
0.33.0 |
zipp |
3.17.0 |
Installed R libraries
R libraries are installed from the Posit Package Manager CRAN snapshot.
Library |
Version |
Library |
Version |
Library |
Version |
|---|---|---|---|---|---|
arrow |
16.1.0 |
askpass |
1.2.0 |
assertthat |
0.2.1 |
backports |
1.5.0 |
base |
4.4.0 |
base64enc |
0.1-3 |
bigD |
0.2.0 |
bit |
4.0.5 |
bit64 |
4.0.5 |
bitops |
1.0-8 |
blob |
1.2.4 |
boot |
1.3-30 |
brew |
1.0-10 |
brio |
1.1.5 |
broom |
1.0.6 |
bslib |
0.8.0 |
cachem |
1.1.0 |
callr |
3.7.6 |
caret |
6.0-94 |
cellranger |
1.1.0 |
chron |
2.3-61 |
class |
7.3-22 |
cli |
3.6.3 |
clipr |
0.8.0 |
clock |
0.7.1 |
cluster |
2.1.6 |
codetools |
0.2-20 |
colorspace |
2.1-1 |
commonmark |
1.9.1 |
compiler |
4.4.0 |
config |
0.3.2 |
conflicted |
1.2.0 |
cpp11 |
0.4.7 |
crayon |
1.5.3 |
credentials |
2.0.1 |
curl |
5.2.1 |
data.table |
1.15.4 |
datasets |
4.4.0 |
DBI |
1.2.3 |
dbplyr |
2.5.0 |
desc |
1.4.3 |
devtools |
2.4.5 |
diagram |
1.6.5 |
diffobj |
0.3.5 |
digest |
0.6.36 |
downlit |
0.4.4 |
dplyr |
1.1.4 |
dtplyr |
1.3.1 |
e1071 |
1.7-14 |
ellipsis |
0.3.2 |
evaluate |
0.24.0 |
fansi |
1.0.6 |
farver |
2.1.2 |
fastmap |
1.2.0 |
fontawesome |
0.5.2 |
forcats |
1.0.0 |
foreach |
1.5.2 |
foreign |
0.8-86 |
forge |
0.2.0 |
fs |
1.6.4 |
future |
1.34.0 |
future.apply |
1.11.2 |
gargle |
1.5.2 |
generics |
0.1.3 |
gert |
2.1.0 |
ggplot2 |
3.5.1 |
gh |
1.4.1 |
git2r |
0.33.0 |
gitcreds |
0.1.2 |
glmnet |
4.1-8 |
globals |
0.16.3 |
glue |
1.7.0 |
googledrive |
2.1.1 |
googlesheets4 |
1.1.1 |
gower |
1.0.1 |
graphics |
4.4.0 |
grDevices |
4.4.0 |
grid |
4.4.0 |
gridExtra |
2.3 |
gsubfn |
0.7 |
gt |
0.11.0 |
gtable |
0.3.5 |
hardhat |
1.4.0 |
haven |
2.5.4 |
highr |
0.11 |
hms |
1.1.3 |
htmltools |
0.5.8.1 |
htmlwidgets |
1.6.4 |
httpuv |
1.6.15 |
httr |
1.4.7 |
httr2 |
1.0.2 |
ids |
1.0.1 |
ini |
0.3.1 |
ipred |
0.9-15 |
isoband |
0.2.7 |
iterators |
1.0.14 |
jquerylib |
0.1.4 |
jsonlite |
1.8.8 |
juicyjuice |
0.1.0 |
KernSmooth |
2.23-22 |
knitr |
1.48 |
labeling |
0.4.3 |
later |
1.3.2 |
lattice |
0.22-5 |
lava |
1.8.0 |
lifecycle |
1.0.4 |
listenv |
0.9.1 |
lubridate |
1.9.3 |
magrittr |
2.0.3 |
markdown |
1.13 |
MASS |
7.3-60.0.1 |
Matrix |
1.6-5 |
memoise |
2.0.1 |
methods |
4.4.0 |
mgcv |
1.9-1 |
mime |
0.12 |
miniUI |
0.1.1.1 |
mlflow |
2.14.1 |
ModelMetrics |
1.2.2.2 |
modelr |
0.1.11 |
munsell |
0.5.1 |
nlme |
3.1-165 |
nnet |
7.3-19 |
numDeriv |
2016.8-1.1 |
openssl |
2.2.0 |
parallel |
4.4.0 |
parallelly |
1.38.0 |
pillar |
1.9.0 |
pkgbuild |
1.4.4 |
pkgconfig |
2.0.3 |
pkgdown |
2.1.0 |
pkgload |
1.4.0 |
plogr |
0.2.0 |
plyr |
1.8.9 |
praise |
1.0.0 |
prettyunits |
1.2.0 |
pROC |
1.18.5 |
processx |
3.8.4 |
prodlim |
2024.06.25 |
profvis |
0.3.8 |
progress |
1.2.3 |
progressr |
0.14.0 |
promises |
1.3.0 |
proto |
1.0.0 |
proxy |
0.4-27 |
ps |
1.7.7 |
purrr |
1.0.2 |
R6 |
2.5.1 |
ragg |
1.3.2 |
randomForest |
4.7-1.1 |
rappdirs |
0.3.3 |
rcmdcheck |
1.4.0 |
RColorBrewer |
1.1-3 |
Rcpp |
1.0.13 |
RcppEigen |
0.3.4.0.0 |
reactable |
0.4.4 |
reactR |
0.6.0 |
readr |
2.1.5 |
readxl |
1.4.3 |
recipes |
1.1.0 |
rematch |
2.0.0 |
rematch2 |
2.1.2 |
remotes |
2.5.0 |
reprex |
2.1.1 |
reshape2 |
1.4.4 |
rlang |
1.1.4 |
rmarkdown |
2.27 |
RODBC |
1.3-23 |
roxygen2 |
7.3.2 |
rpart |
4.1.23 |
rprojroot |
2.0.4 |
Rserve |
1.8-13 |
RSQLite |
2.3.7 |
rstudioapi |
0.16.0 |
rversions |
2.1.2 |
rvest |
1.0.4 |
sass |
0.4.9 |
scales |
1.3.0 |
selectr |
0.4-2 |
sessioninfo |
1.2.2 |
shape |
1.4.6.1 |
shiny |
1.9.1 |
sourcetools |
0.1.7-1 |
sparklyr |
1.8.6 |
spatial |
7.3-17 |
splines |
4.4.0 |
sqldf |
0.4-11 |
SQUAREM |
2021.1 |
stats |
4.4.0 |
stats4 |
4.4.0 |
stringi |
1.8.4 |
stringr |
1.5.1 |
survival |
3.6-4 |
swagger |
5.17.14.1 |
sys |
3.4.2 |
systemfonts |
1.1.0 |
tcltk |
4.4.0 |
testthat |
3.2.1.1 |
textshaping |
0.4.0 |
tibble |
3.2.1 |
tidyr |
1.3.1 |
tidyselect |
1.2.1 |
tidyverse |
2.0.0 |
timechange |
0.3.0 |
timeDate |
4032.109 |
tinytex |
0.52 |
tools |
4.4.0 |
tzdb |
0.4.0 |
urlchecker |
1.0.1 |
usethis |
3.0.0 |
utf8 |
1.2.4 |
utils |
4.4.0 |
uuid |
1.2-1 |
V8 |
4.4.2 |
vctrs |
0.6.5 |
viridisLite |
0.4.2 |
vroom |
1.6.5 |
waldo |
0.5.2 |
whisker |
0.4.1 |
withr |
3.0.1 |
xfun |
0.46 |
xml2 |
1.3.6 |
xopen |
1.0.1 |
xtable |
1.8-4 |
yaml |
2.3.10 |
zeallot |
0.1.0 |
zip |
2.3.1 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
|---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.638 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.638 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.638 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.638 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.638 |
com.amazonaws |
aws-java-sdk-config |
1.12.638 |
com.amazonaws |
aws-java-sdk-core |
1.12.638 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.638 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.638 |
com.amazonaws |
aws-java-sdk-directory |
1.12.638 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.638 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.638 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.638 |
com.amazonaws |
aws-java-sdk-efs |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.638 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.638 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.638 |
com.amazonaws |
aws-java-sdk-emr |
1.12.638 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.638 |
com.amazonaws |
aws-java-sdk-glue |
1.12.638 |
com.amazonaws |
aws-java-sdk-iam |
1.12.638 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.638 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.638 |
com.amazonaws |
aws-java-sdk-kms |
1.12.638 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.638 |
com.amazonaws |
aws-java-sdk-logs |
1.12.638 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.638 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.638 |
com.amazonaws |
aws-java-sdk-rds |
1.12.638 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.638 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.638 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.638 |
com.amazonaws |
aws-java-sdk-ses |
1.12.638 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.638 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.638 |
com.amazonaws |
aws-java-sdk-sns |
1.12.638 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.638 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.638 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.638 |
com.amazonaws |
aws-java-sdk-sts |
1.12.638 |
com.amazonaws |
aws-java-sdk-support |
1.12.638 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.638 |
com.amazonaws |
jmespath-java |
1.12.638 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
databricks-sdk-java |
0.27.0 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.15.2 |
com.fasterxml.jackson.core |
jackson-core |
2.15.2 |
com.fasterxml.jackson.core |
jackson-databind |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.15.2 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-yaml |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.15.2 |
com.fasterxml.jackson.datatype |
jackson-datatype-jsr310 |
2.16.0 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.15.2 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.15.2 |
com.github.ben-manes.caffeine |
caffeine |
2.9.3 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java |
1.1-natives |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1-natives |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64 |
1.1-natives |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64 |
1.1-natives |
com.github.luben |
zstd-jni |
1.5.5-4 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.10.1 |
com.google.crypto.tink |
tink |
1.9.0 |
com.google.errorprone |
error_prone_annotations |
2.10.0 |
com.google.flatbuffers |
flatbuffers-java |
23.5.26 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
3.25.1 |
com.helger |
profiler |
1.1.1 |
com.ibm.icu |
icu4j |
75.1 |
com.jcraft |
jsch |
0.1.55 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.microsoft.sqlserver |
mssql-jdbc |
11.2.2.jre8 |
com.ning |
compress-lzf |
1.1.2 |
com.sun.mail |
javax.mail |
1.5.2 |
com.sun.xml.bind |
jaxb-core |
2.2.11 |
com.sun.xml.bind |
jaxb-impl |
2.2.11 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.4.3 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.uber |
h3 |
3.7.3 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.16.0 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.5 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.13.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
3.0.3 |
dev.ludovic.netlib |
blas |
3.0.3 |
dev.ludovic.netlib |
lapack |
3.0.3 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.27 |
io.delta |
delta-sharing-client_2.12 |
1.2.0 |
io.dropwizard.metrics |
metrics-annotation |
4.2.19 |
io.dropwizard.metrics |
metrics-core |
4.2.19 |
io.dropwizard.metrics |
metrics-graphite |
4.2.19 |
io.dropwizard.metrics |
metrics-healthchecks |
4.2.19 |
io.dropwizard.metrics |
metrics-jetty9 |
4.2.19 |
io.dropwizard.metrics |
metrics-jmx |
4.2.19 |
io.dropwizard.metrics |
metrics-json |
4.2.19 |
io.dropwizard.metrics |
metrics-jvm |
4.2.19 |
io.dropwizard.metrics |
metrics-servlets |
4.2.19 |
io.netty |
netty-all |
4.1.108.Final |
io.netty |
netty-buffer |
4.1.108.Final |
io.netty |
netty-codec |
4.1.108.Final |
io.netty |
netty-codec-http |
4.1.108.Final |
io.netty |
netty-codec-http2 |
4.1.108.Final |
io.netty |
netty-codec-socks |
4.1.108.Final |
io.netty |
netty-common |
4.1.108.Final |
io.netty |
netty-handler |
4.1.108.Final |
io.netty |
netty-handler-proxy |
4.1.108.Final |
io.netty |
netty-resolver |
4.1.108.Final |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-linux-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-aarch_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-osx-x86_64 |
io.netty |
netty-tcnative-boringssl-static |
2.0.61.Final-windows-x86_64 |
io.netty |
netty-tcnative-classes |
2.0.61.Final |
io.netty |
netty-transport |
4.1.108.Final |
io.netty |
netty-transport-classes-epoll |
4.1.108.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.108.Final |
io.netty |
netty-transport-native-epoll |
4.1.108.Final |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-aarch_64 |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-riscv64 |
io.netty |
netty-transport-native-epoll |
4.1.108.Final-linux-x86_64 |
io.netty |
netty-transport-native-kqueue |
4.1.108.Final-osx-aarch_64 |
io.netty |
netty-transport-native-kqueue |
4.1.108.Final-osx-x86_64 |
io.netty |
netty-transport-native-unix-common |
4.1.108.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.12.1 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.3 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.9.3 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.10.11 |
org.apache.ant |
ant-jsch |
1.10.11 |
org.apache.ant |
ant-launcher |
1.10.11 |
org.apache.arrow |
arrow-format |
15.0.0 |
org.apache.arrow |
arrow-memory-core |
15.0.0 |
org.apache.arrow |
arrow-memory-netty |
15.0.0 |
org.apache.arrow |
arrow-vector |
15.0.0 |
org.apache.avro |
avro |
1.11.3 |
org.apache.avro |
avro-ipc |
1.11.3 |
org.apache.avro |
avro-mapred |
1.11.3 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.23.0 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.10.0 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.datasketches |
datasketches-java |
3.1.0 |
org.apache.datasketches |
datasketches-memory |
2.0.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-runtime |
3.3.6 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.8.1 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.14 |
org.apache.httpcomponents |
httpcore |
4.4.16 |
org.apache.ivy |
ivy |
2.5.1 |
org.apache.logging.log4j |
log4j-1.2-api |
2.22.1 |
org.apache.logging.log4j |
log4j-api |
2.22.1 |
org.apache.logging.log4j |
log4j-core |
2.22.1 |
org.apache.logging.log4j |
log4j-layout-template-json |
2.22.1 |
org.apache.logging.log4j |
log4j-slf4j2-impl |
2.22.1 |
org.apache.orc |
orc-core |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-mapreduce |
1.9.2-shaded-protobuf |
org.apache.orc |
orc-shims |
1.9.2 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.ws.xmlschema |
xmlschema-core |
2.3.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.23 |
org.apache.yetus |
audience-annotations |
0.13.0 |
org.apache.zookeeper |
zookeeper |
3.9.2 |
org.apache.zookeeper |
zookeeper-jute |
3.9.2 |
org.checkerframework |
checker-qual |
3.31.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.collections |
eclipse-collections |
11.1.0 |
org.eclipse.collections |
eclipse-collections-api |
11.1.0 |
org.eclipse.jetty |
jetty-client |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-continuation |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-http |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-io |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-jndi |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-plus |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-proxy |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-security |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-server |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlet |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-servlets |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-webapp |
9.4.52.v20230823 |
org.eclipse.jetty |
jetty-xml |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.52.v20230823 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.52.v20230823 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.40 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.40 |
org.glassfish.jersey.core |
jersey-client |
2.40 |
org.glassfish.jersey.core |
jersey-common |
2.40 |
org.glassfish.jersey.core |
jersey-server |
2.40 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.40 |
org.hibernate.validator |
hibernate-validator |
6.1.7.Final |
org.ini4j |
ini4j |
0.5.4 |
org.javassist |
javassist |
3.29.2-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mlflow |
mlflow-spark_2.12 |
2.9.1 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.6.1 |
org.roaringbitmap |
RoaringBitmap |
0.9.45-databricks |
org.roaringbitmap |
shims |
0.9.45-databricks |
org.rocksdb |
rocksdbjni |
9.2.1 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.15 |
org.scala-lang |
scala-library_2.12 |
2.12.15 |
org.scala-lang |
scala-reflect_2.12 |
2.12.15 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.11.0 |
org.scala-lang.modules |
scala-java8-compat_2.12 |
0.9.1 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.2.16 |
org.scalanlp |
breeze-macros_2.12 |
2.1.0 |
org.scalanlp |
breeze_2.12 |
2.1.0 |
org.scalatest |
scalatest-compatible |
3.2.16 |
org.scalatest |
scalatest-core_2.12 |
3.2.16 |
org.scalatest |
scalatest-diagrams_2.12 |
3.2.16 |
org.scalatest |
scalatest-featurespec_2.12 |
3.2.16 |
org.scalatest |
scalatest-flatspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-freespec_2.12 |
3.2.16 |
org.scalatest |
scalatest-funspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-funsuite_2.12 |
3.2.16 |
org.scalatest |
scalatest-matchers-core_2.12 |
3.2.16 |
org.scalatest |
scalatest-mustmatchers_2.12 |
3.2.16 |
org.scalatest |
scalatest-propspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-refspec_2.12 |
3.2.16 |
org.scalatest |
scalatest-shouldmatchers_2.12 |
3.2.16 |
org.scalatest |
scalatest-wordspec_2.12 |
3.2.16 |
org.scalatest |
scalatest_2.12 |
3.2.16 |
org.slf4j |
jcl-over-slf4j |
2.0.7 |
org.slf4j |
jul-to-slf4j |
2.0.7 |
org.slf4j |
slf4j-api |
2.0.7 |
org.slf4j |
slf4j-simple |
1.7.25 |
org.threeten |
threeten-extra |
1.7.1 |
org.tukaani |
xz |
1.9 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.1.3.Final |
org.xerial |
sqlite-jdbc |
3.42.0.0 |
org.xerial.snappy |
snappy-java |
1.1.10.3 |
org.yaml |
snakeyaml |
2.0 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.cryptools |
AmazonCorrettoCryptoProvider |
1.6.2-linux-x86_64 |
stax |
stax-api |
1.0.1 |