Pular para o conteúdo principal

Databricks Runtime 16.4 LTS (Beta)

Beta

Databricks Runtime 16.4 LTS is in Beta. The contents of the supported environments might change during the Beta. Changes can include the list of packages or versions of installed packages.

The following release notes provide information about Databricks Runtime 16.4 LTS, powered by Apache Spark 3.5.2.

Databricks released this beta version in April 2025.

dica

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes​

Delta tables: "DESCRIBE DETAIL table" will now show the clusterByAuto status of the table​

DESCRIBE DETAIL {table} will now show the clusterByAuto status of the table (true or false) next to the current clustering columns. For more details on clusterByAuto, see:Automatic liquid clustering.

Fix to respect options for data source cached plans​

This update ensures table reads respect options set for all data source plans when cached, not just the first cached table read.

Previously, data source table reads cached the first plan but failed to account for different options in subsequent queries.

For example, the following query:

spark.sql("CREATE TABLE t(a string, b string) USING CSV".stripMargin)
spark.sql("INSERT INTO TABLE t VALUES ('a;b', 'c')")

spark.sql("SELECT * FROM t").show()
spark.sql("SELECT * FROM t WITH ('delimiter' = ';')")

would produce this output:

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

With this fix, it now returns the expected output:

+----+----+
|col1|col2|
+----+----+
| a;b| c  |
+----+----+

+----+----+
|col1|col2|
+----+----+
| a  | b,c|
+----+----+

If your workloads have dependencies on the previous incorrect behavior, you may see different results after this change.

Fix for grouping on aliased integer literals broke for certain operations​

Grouping expressions on an aliased integer literal was previously broken for certain operations like MERGE INTO. For example, this expression would return GROUP_BY_POS_OUT_OF_RANGE since the value (val) would be replaced with 202001:

SQL
merge into t
using
(select 202001 as val, count(current_date) as total_count group by val) on 1=1
when not matched then insert (id, name)  values (val, total_count)

This has been fixed. To mitigate the issue in your existing queries, make sure that the constants you are using are not equal to the column position that must be in the grouping expressions.

Moved redaction rule from analyzer to optimizer​

Previously, DataFrames could create tables that contained redacted values when valid SECRET SQL functions were used. This change removes redaction when saving DataFrames with valid secret access to a table, and the redaction rule has moved from the analyzer to the optimizer.

variant_get and get_json_object now consider leading spaces in paths in Apache Spark​

Prior to this change, leading whitespaces and tabs in paths in the variant_get and get_json_object expressions were being ignored with Photon disabled. For example, select get_json_object('{" key": "value"}', '$[' key']') would not be effective in extracting the value of " key". However, users will be able to extract such keys.

Enable flag to disallow disabling source materialization for MERGE operations​

Previously, users could disable source materialization in MERGE by setting merge.materializeSource to none. With the new flag enabled, this will be forbidden and cause an error. Databricks plans to enable the flag only for customers who haven't used this configuration flag before, so no customer should notice any change in behavior.

Move partition metadata log enablement anchor to table​

The partition metadata log feature has been changed so that once a table is created with spark.databricks.nonDelta.partitionLog.enabled = true, you can anchor it to a table so a cluster doesn't set spark.databricks.nonDelta.partitionLog.enabled = true for all tables processed by the cluster.

Update snowflake-jdbc 3.16.1 -> 3.22.0​

Updated the dependency snowflake-jdbc from 3.16.1 to 3.22.0. This may impact users if they directly use the l3.16.1 version of the library.

Downgrade Json4s from 4.0.7 to 3.7.0-M11 in DBR 16.2 and 16.3​

Customers cannot use databricks-connect 16.1+ and Apache Spark™ 3.5.x together in the same application because of significant discrepancies in API behavior between Json4s version 3.7.0-M11 and version 4.0.7. To address this, Databricks has downgraded Json4s to 3.7.0-M11.

New features and improvements​

New listagg and string_agg functions​

Starting with this release you can use the listagg or string_agg functions to aggregate STRING and BINARY values within a group. See string_agg for more details.

Library upgrades​

  • Upgraded Python libraries:
  • Upgraded R libraries:
  • Upgraded Java libraries:
    • io.delta.delta-sharing-client_2.12 from 1.2.3 to 1.2.4
    • org.json4s.json4s-ast_2.12 from 4.0.7 to 3.7.0-M11
    • org.json4s.json4s-core_2.12 from 4.0.7 to 3.7.0-M11
    • org.json4s.json4s-jackson_2.12 from 4.0.7 to 3.7.0-M11
    • org.json4s.json4s-scalap_2.12 from 4.0.7 to 3.7.0-M11

Apache Spark​

Databricks Runtime 16.4 LTS includes Apache Spark 3.5.2. This release includes all Spark fixes and improvements included in Databricks Runtime 16.3, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-51762] Fix Resolution of Views in Single-Pass Analyzer When Bridging is Enabled
  • [SPARK-51716] Support serializing Variant to XML
  • [SPARK-51503] Support Variant type in XML scan
  • [SPARK-47895] Revert "[SQL] group by alias should be idempotent"
  • [SPARK-51752] Enable rCTE referencing from within a CTE
  • [SPARK-51747] [SQL] Data source cached plan should respect options
  • [SPARK-51758] Apply late record filtering based on watermark only if timeMode is passed as EventTime to the transformWithState operator
  • [SPARK-51576] Prohibit ANSI cast from array<variant, containsNull = false> to array<_, containsNull = false>
  • [SPARK-51776] Fix logging in single-pass Analyzer
  • [SPARK-51717] Fix SST mismatch corruption that can happen for second snapshot created for a new query
  • [SPARK-51751] Fix multiple rCTEs for one WITH statement that reference each other
  • [SPARK-51712] [SQL] Swallow non-fatal Throwables when resolving tables/views in spark.catalog.listTables()
  • [SPARK-51646] Fix propagating collation in views with default collation
  • [SPARK-51690] Change the protocol of ListState.put()/get()/appendList() from Arrow to simple custom protocol
  • [SPARK-51732] Apply rpad on attributes with same ExprId if they need to be deduplicated
  • [SPARK-51724] RocksDB StateStore's lineage manager should be synchronized
  • [SPARK-51720] Add Cross Join as legal in recursion of Recursive CTE
  • [SPARK-51682] State Store Checkpoint V2 should handle offset log ahead of commit log correctly
  • [SPARK-51721] Change default value of ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED flag
  • [SPARK-51685] Excessive Info logging from RocksDb operations causing too big executor stderr files
  • [SPARK-51645] Fix CREATE OR REPLACE TABLE ... DEFAULT COLLATION ... query
  • [SPARK-51298] Support variant in CSV scan
  • [SPARK-51271] Add filter pushdown API to Python Data Sources
  • [SPARK-51667] [PYTHON] Disable Nagle's algorithm (via TCP_NODELAY = true) in TWS + PySpark for python with regards to state server
  • [SPARK-51668] Report metrics for failed writes to V2 data sources
  • [SPARK-51187] Implement the graceful deprecation of incorrect config introduced in SPARK-49699
  • [SPARK-51700] Fix incorrect logging when no files are eligible for deletion in RocksDBFileManager
  • [SPARK-51655] Fix metric collection in UnionLoopExec and add test
  • [SPARK-51657] [SC-192703] UTF8_BINARY default table collation shown by default in Desc As JSON (v1)
  • [SPARK-51206] [CONNECT] Move Arrow conversion helpers out of Spark Connect
  • [SPARK-51566] Python UDF traceback improvement
  • [SPARK-51675] Fix col family creation after opening local DB to avoid snapshot creation, if not necessary
  • [SPARK-50892] Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries
  • [SPARK-51670] Refactor Intersect and Except to follow Union example to reuse in single-pass Analyzer
  • [SPARK-47895] [SQL] group by alias should be idempotent
  • [SPARK-51614] [SQL] Introduce ResolveUnresolvedHaving rule in the Analyzer
  • [SPARK-51662] Make OrcFileFormat comparable
  • [SPARK-51374] Revert "[SC-190311][core] Switch to Using java.util.Map in Logging APIs"
  • [SPARK-51652] Refactor SetOperation computation out to reuse it in the single-pass Analyzer
  • [SPARK-51622] Titling sections on ExecutionPage
  • [SPARK-51586] Initialize input partitions independent of columnar support in continuous mode
  • [SPARK-51070] Use scala.collection.Set instead of Set in ValidateExternalType
  • [SPARK-51559] Make max broadcast table size configurable
  • [SPARK-51496] CaseInsensitiveStringMap comparison should ignore case
  • [SPARK-50416] A more portable terminal / pipe test needed for bin/load-spark-env.sh
  • [SPARK-51509] [UI] Make Spark Master Environment page support filters
  • [SPARK-51374] Switch to Using java.util.Map in Logging APIs
  • [SPARK-50820] DSv2: Conditional nullification of metadata columns in DML
  • [SPARK-48922] Avoid redundant array transform of identical expression for map type
  • [SPARK-51446] Improve the codecNameMap for the compression codec
  • [SPARK-51341] Cancel time task with suitable way.
  • [SPARK-51452] Improve Thread dump table search
  • [SPARK-51444] Remove the unreachable if branch from TaskSchedulerImpl#statusUpdate
  • [SPARK-51401] Change ExplainUtils.generateFieldString to directly call QueryPlan.generateFieldString
  • [SPARK-49507] Fix the case issue after enabling metastorePartitionPruningFastFallback
  • [SPARK-51506] [SS] Do not enforce users to implement close() in TransformWithStateInPandas
  • [SPARK-51624] Propagate GetStructField metadata in CreateNamedStruct.dataType
  • [SPARK-51201] Make Partitioning Hints support byte and short values
  • [SPARK-51186] Add StreamingPythonRunnerInitializationException to PySpark base exception
  • [SPARK-50286] Correctly propagate SQL options to WriteBuilder
  • [SPARK-51616] Run CollationTypeCasts before ResolveAliases and ResolveAggregateFunctions
  • [SPARK-51023] Log remote address on RPC exception
  • [SPARK-51087] [CONNECT] Raise a warning when memory-profiler is not installed for memory profiling
  • [SPARK-51073] Remove Unstable from SparkSessionExtensionsProvider trait
  • [SPARK-51471] RatePerMicroBatchStream - classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset
  • [SPARK-51062] Fix assertSchemaEqual to compare decimal precision and scale
  • [SPARK-51612] Display Spark confs set at view creation in Desc As Json
  • [SPARK-51573] Fix Streaming State Checkpoint v2 checkpointInfo race condition
  • [SPARK-51252] Add instance metrics for last uploaded snapshot version in HDFS State Stores
  • [SPARK-51552] [SQL] Disallow temporary variables in persisted views when under identifier
  • [SPARK-51625] Command in CTE relations should trigger inline
  • [SPARK-51584] Add rule that pushes Project through Offset and Suite that tests it
  • [SPARK-51593] Refactor QueryExecutionMetering instantiation
  • [SPARK-49082] Support widening Date to TimestampNTZ in Avro reader
  • [SPARK-50220] Support listagg in PySpark
  • [SPARK-51581] [SQL] Use nonEmpty/isEmpty for empty check for explicit Iterable
  • [SPARK-42746] Fix in error class classification for new exceptions with listagg
  • [SPARK-51580] Throw proper user facing error message when lambda function is out of place in HigherOrderFunction
  • [SPARK-51589] Fix small bug failing to check for aggregate functions in |> SELECT
  • [SPARK-50821] Upgrade Py4J from 0.10.9.8 to 0.10.9.9
  • [SPARK-42746] Implement LISTAGG function
  • [SPARK-48399] Teradata: ByteType should map to BYTEINT instead of BYTE(binary)
  • [SPARK-43221] Host local block fetching should use a block status of a block stored on disk
  • [SPARK-51467] Make tables of the environment page filterable
  • [SPARK-51569] Don't reinstantiate InSubquery in InTypeCoercion if there are no type changes
  • [SPARK-51280] Improve RESPONSE_ALREADY_RECEIVED error class
  • [SPARK-51525] Collation field for Desc As JSON StringType
  • [SPARK-51565] Support SQL parameters in window frame clause
  • [SPARK-51544] Add only unique and necessary metadata columns
  • [SPARK-49349] Improve error message for LCA with Generate
  • [SPARK-51418] Fix DataSource PARTITON TABLE w/ Hive type incompatible partition columns
  • [SPARK-51269] Simplify AvroCompressionCodec by removing defaultCompressionLevel
  • [SPARK-50618] [SQL] Make DataFrameReader and DataStreamReader leverage the analyzer more
  • [SPARK-51097] Re-introduce RocksDB state store's last uploaded snapshot version instance metrics
  • [SPARK-51321] Add rpad and lpad support for PostgresDialect and MsSQLServerDialect expression pushdown
  • [SPARK-51428] Reassign Aliases for collated expression trees deterministically
  • [SPARK-51528] Don't compare hash codes directly in ProtobufCatalystDataConversionSuite
  • [SPARK-51443] Fix singleVariantColumn in DSv2 and readStream.
  • [SPARK-51397] Fix maintenance pool shutdown handling issue causing long test times
  • [SPARK-51522] Disable lazy union children test for single-pass Analyzer
  • [SPARK-51079] Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
  • [SPARK-51112] [CONNECT] Avoid using pyarrow's to_pandas on an empty table
  • [SPARK-50855] Follow up on [SC-190361] Fix post-merge tests failure in isolated code paths
  • [SPARK-51438] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable
  • [SPARK-50855] Spark Connect Support for TransformWithState In Scala
  • [SPARK-51208] ColumnDefinition.toV1Column should preserve EXISTS_DEFAULT resolution
  • [SPARK-51468] Revert "From json/xml should not change collations in the given schema"
  • [SPARK-51453] [SC-190922] [SQL] AssertTrue uses toPrettySQL instead of simpleString
  • [SPARK-51409] Add error classification in the changelog writer creation path
  • [SPARK-51440] classify the NPE when null topic field value is in kafka message data and there is no topic option
  • [SPARK-51425] Add client API to set custom operation_id
  • [SPARK-50652] Add checks to RocksDB V2 backward compatibility
  • [SPARK-49164] Fix not NullSafeEqual in predicate of SQL query in JDBC Relation
  • [SPARK-50880] Add a new visitBinaryComparison method to V2ExpressionSQLBuilder
  • [SPARK-50838] Performs additional checks inside recursive CTEs to throw an error if forbidden case is encountered
  • [SPARK-51307] locationUri in CatalogStorageFormat shall be decoded for display
  • [SPARK-51079] Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
  • [SPARK-50792] Format binary data as a binary literal in JDBC.
  • [SPARK-50596] Upgrade Py4J from 0.10.9.7 to 0.10.9.8

Databricks ODBC/JDBC driver support​

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment​

  • Operating System: Ubuntu 24.04.2 LTS
  • Java: Zulu17.54+21-CA
  • Scala: 2.12.15
  • Python: 3.12.3
  • R: 4.4.0
  • Delta Lake: 3.3.0

Installed Python libraries​

Library

Version

Library

Version

Library

Version

annotated-types

0.7.0

asttokens

2.0.5

astunparse

1.6.3

autocommand

2.2.2

azure-core

1.31.0

azure-storage-blob

12.23.0

azure-storage-file-datalake

12.17.0

backports.tarfile

1.2.0

black

24.4.2

blinker

1.7.0

boto3

1.34.69

botocore

1.34.69

cachetools

5.3.3

certifi

2024.6.2

cffi

1.16.0

chardet

4.0.0

charset-normalizer

2.0.4

click

8.1.7

cloudpickle

2.2.1

comm

0.2.1

contourpy

1.2.0

cryptography

42.0.5

cycler

0.11.0

Cython

3.0.11

databricks-sdk

0.30.0

dbus-python

1.3.2

debugpy

1.6.7

decorator

5.1.1

Deprecated

1.2.14

distlib

0.3.8

docstring-to-markdown

0.11

executing

0.8.3

facets-overview

1.1.1

filelock

3.15.4

fonttools

4.51.0

gitdb

4.0.11

GitPython

3.1.37

google-api-core

2.20.0

google-auth

2.35.0

google-cloud-core

2.4.1

google-cloud-storage

2.18.2

google-crc32c

1.6.0

google-resumable-media

2.7.2

googleapis-common-protos

1.65.0

grpcio

1.60.0

grpcio-status

1.60.0

httplib2

0.20.4

idna

3.7

importlib-metadata

6.0.0

importlib_resources

6.4.0

inflect

7.3.1

ipyflow-core

0.0.201

ipykernel

6.28.0

ipython

8.25.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

isodate

0.6.1

jaraco.context

5.3.0

jaraco.functools

4.0.1

jaraco.text

3.12.1

jedi

0.19.1

jmespath

1.0.1

joblib

1.4.2

jupyter_client

8.6.0

jupyter_core

5.7.2

kiwisolver

1.4.4

launchpadlib

1.11.0

lazr.restfulclient

0.14.6

lazr.uri

1.0.6

matplotlib

3.8.4

matplotlib-inline

0.1.6

mccabe

0.7.0

mlflow-skinny

2.19.0

more-itertools

10.3.0

mypy

1.10.0

mypy-extensions

1.0.0

nest-asyncio

1.6.0

nodeenv

1.9.1

numpy

1.26.4

oauthlib

3.2.2

opentelemetry-api

1.27.0

opentelemetry-sdk

1.27.0

opentelemetry-semantic-conventions

0.48b0

packaging

24.1

pandas

1.5.3

parso

0.8.3

pathspec

0.10.3

patsy

0.5.6

pexpect

4.8.0

pillow

10.3.0

pip

24.2

platformdirs

3.10.0

plotly

5.22.0

pluggy

1.0.0

prompt-toolkit

3.0.43

proto-plus

1.24.0

protobuf

4.24.1

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

15.0.2

pyasn1

0.4.8

pyasn1-modules

0.2.8

pyccolo

0.0.65

pycparser

2.21

pydantic

2.8.2

pydantic_core

2.20.1

pyflakes

3.2.0

Pygments

2.15.1

PyGObject

3.48.2

PyJWT

2.7.0

pyodbc

5.0.1

pyparsing

3.0.9

pyright

1.1.294

python-dateutil

2.9.0.post0

python-lsp-jsonrpc

1.1.2

python-lsp-server

1.10.0

pytoolconfig

1.2.6

pytz

2024.1

PyYAML

6.0.1

pyzmq

25.1.2

requests

2.32.2

rope

1.12.0

rsa

4.9

s3transfer

0.10.2

scikit-learn

1.4.2

scipy

1.13.1

seaborn

0.13.2

setuptools

74.0.0

six

1.16.0

smmap

5.0.0

sqlparse

0.5.1

ssh-import-id

5.11

stack-data

0.2.0

statsmodels

0.14.2

tenacity

8.2.2

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.4.1

traitlets

5.14.3

typeguard

4.3.0

types-protobuf

3.20.3

types-psutil

5.9.0

types-pytz

2023.3.1.1

types-PyYAML

6.0.0

types-requests

2.31.0.0

types-setuptools

68.0.0.0

types-six

1.16.0

types-urllib3

1.26.25.14

typing_extensions

4.11.0

ujson

5.10.0

unattended-upgrades

0.1

urllib3

1.26.16

virtualenv

20.26.2

wadllib

1.3.6

wcwidth

0.2.5

whatthepatch

1.0.2

wheel

0.43.0

wrapt

1.14.1

yapf

0.33.0

zipp

3.17.0

Installed R libraries​

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20:

Library

Version

Library

Version

Library

Version

arrow

16.1.0

askpass

1.2.0

assertthat

0.2.1

backports

1.5.0

base

4.4.0

base64enc

0.1-3

bigD

0.2.0

bit

4.0.5

bit64

4.0.5

bitops

1.0-8

blob

1.2.4

boot

1.3-30

brew

1.0-10

brio

1.1.5

broom

1.0.6

bslib

0.8.0

cachem

1.1.0

callr

3.7.6

caret

6.0-94

cellranger

1.1.0

chron

2.3-61

class

7.3-22

cli

3.6.3

clipr

0.8.0

clock

0.7.1

cluster

2.1.6

codetools

0.2-20

colorspace

2.1-1

commonmark

1.9.1

compiler

4.4.0

config

0.3.2

conflicted

1.2.0

cpp11

0.4.7

crayon

1.5.3

credentials

2.0.1

curl

5.2.1

data.table

1.15.4

datasets

4.4.0

DBI

1.2.3

dbplyr

2.5.0

desc

1.4.3

devtools

2.4.5

diagram

1.6.5

diffobj

0.3.5

digest

0.6.36

downlit

0.4.4

dplyr

1.1.4

dtplyr

1.3.1

e1071

1.7-14

ellipsis

0.3.2

evaluate

0.24.0

fansi

1.0.6

farver

2.1.2

fastmap

1.2.0

fontawesome

0.5.2

forcats

1.0.0

foreach

1.5.2

foreign

0.8-86

forge

0.2.0

fs

1.6.4

future

1.34.0

future.apply

1.11.2

gargle

1.5.2

generics

0.1.3

gert

2.1.0

ggplot2

3.5.1

gh

1.4.1

git2r

0.33.0

gitcreds

0.1.2

glmnet

4.1-8

globals

0.16.3

glue

1.7.0

googledrive

2.1.1

googlesheets4

1.1.1

gower

1.0.1

graphics

4.4.0

grDevices

4.4.0

grid

4.4.0

gridExtra

2.3

gsubfn

0.7

gt

0.11.0

gtable

0.3.5

hardhat

1.4.0

haven

2.5.4

highr

0.11

hms

1.1.3

htmltools

0.5.8.1

htmlwidgets

1.6.4

httpuv

1.6.15

httr

1.4.7

httr2

1.0.2

ids

1.0.1

ini

0.3.1

ipred

0.9-15

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.8

juicyjuice

0.1.0

KernSmooth

2.23-22

knitr

1.48

labeling

0.4.3

later

1.3.2

lattice

0.22-5

lava

1.8.0

lifecycle

1.0.4

listenv

0.9.1

lubridate

1.9.3

magrittr

2.0.3

markdown

1.13

MASS

7.3-60.0.1

Matrix

1.6-5

memoise

2.0.1

methods

4.4.0

mgcv

1.9-1

mime

0.12

miniUI

0.1.1.1

mlflow

2.14.1

ModelMetrics

1.2.2.2

modelr

0.1.11

munsell

0.5.1

nlme

3.1-165

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.2.0

parallel

4.4.0

parallelly

1.38.0

pillar

1.9.0

pkgbuild

1.4.4

pkgconfig

2.0.3

pkgdown

2.1.0

pkgload

1.4.0

plogr

0.2.0

plyr

1.8.9

praise

1.0.0

prettyunits

1.2.0

pROC

1.18.5

processx

3.8.4

prodlim

2024.06.25

profvis

0.3.8

progress

1.2.3

progressr

0.14.0

promises

1.3.0

proto

1.0.0

proxy

0.4-27

ps

1.7.7

purrr

1.0.2

R6

2.5.1

ragg

1.3.2

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.13

RcppEigen

0.3.4.0.0

reactable

0.4.4

reactR

0.6.0

readr

2.1.5

readxl

1.4.3

recipes

1.1.0

rematch

2.0.0

rematch2

2.1.2

remotes

2.5.0

reprex

2.1.1

reshape2

1.4.4

rlang

1.1.4

rmarkdown

2.27

RODBC

1.3-23

roxygen2

7.3.2

rpart

4.1.23

rprojroot

2.0.4

Rserve

1.8-13

RSQLite

2.3.7

rstudioapi

0.16.0

rversions

2.1.2

rvest

1.0.4

sass

0.4.9

scales

1.3.0

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6.1

shiny

1.9.1

sourcetools

0.1.7-1

sparklyr

1.8.6

SparkR

3.5.2

spatial

7.3-17

splines

4.4.0

sqldf

0.4-11

SQUAREM

2021.1

stats

4.4.0

stats4

4.4.0

stringi

1.8.4

stringr

1.5.1

survival

3.6-4

swagger

5.17.14.1

sys

3.4.2

systemfonts

1.1.0

tcltk

4.4.0

testthat

3.2.1.1

textshaping

0.4.0

tibble

3.2.1

tidyr

1.3.1

tidyselect

1.2.1

tidyverse

2.0.0

timechange

0.3.0

timeDate

4032.109

tinytex

0.52

tools

4.4.0

tzdb

0.4.0

urlchecker

1.0.1

usethis

3.0.0

utf8

1.2.4

utils

4.4.0

uuid

1.2-1

V8

4.4.2

vctrs

0.6.5

viridisLite

0.4.2

vroom

1.6.5

waldo

0.5.2

whisker

0.4.1

withr

3.0.1

xfun

0.46

xml2

1.3.6

xopen

1.0.1

xtable

1.8-4

yaml

2.3.10

zeallot

0.1.0

zip

2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)​

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.638

com.amazonaws

aws-java-sdk-cloudformation

1.12.638

com.amazonaws

aws-java-sdk-cloudfront

1.12.638

com.amazonaws

aws-java-sdk-cloudhsm

1.12.638

com.amazonaws

aws-java-sdk-cloudsearch

1.12.638

com.amazonaws

aws-java-sdk-cloudtrail

1.12.638

com.amazonaws

aws-java-sdk-cloudwatch

1.12.638

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.638

com.amazonaws

aws-java-sdk-codedeploy

1.12.638

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.638

com.amazonaws

aws-java-sdk-cognitosync

1.12.638

com.amazonaws

aws-java-sdk-config

1.12.638

com.amazonaws

aws-java-sdk-core

1.12.638

com.amazonaws

aws-java-sdk-datapipeline

1.12.638

com.amazonaws

aws-java-sdk-directconnect

1.12.638

com.amazonaws

aws-java-sdk-directory

1.12.638

com.amazonaws

aws-java-sdk-dynamodb

1.12.638

com.amazonaws

aws-java-sdk-ec2

1.12.638

com.amazonaws

aws-java-sdk-ecs

1.12.638

com.amazonaws

aws-java-sdk-efs

1.12.638

com.amazonaws

aws-java-sdk-elasticache

1.12.638

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.638

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.638

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.638

com.amazonaws

aws-java-sdk-emr

1.12.638

com.amazonaws

aws-java-sdk-glacier

1.12.638

com.amazonaws

aws-java-sdk-glue

1.12.638

com.amazonaws

aws-java-sdk-iam

1.12.638

com.amazonaws

aws-java-sdk-importexport

1.12.638

com.amazonaws

aws-java-sdk-kinesis

1.12.638

com.amazonaws

aws-java-sdk-kms

1.12.638

com.amazonaws

aws-java-sdk-lambda

1.12.638

com.amazonaws

aws-java-sdk-logs

1.12.638

com.amazonaws

aws-java-sdk-machinelearning

1.12.638

com.amazonaws

aws-java-sdk-opsworks

1.12.638

com.amazonaws

aws-java-sdk-rds

1.12.638

com.amazonaws

aws-java-sdk-redshift

1.12.638

com.amazonaws

aws-java-sdk-route53

1.12.638

com.amazonaws

aws-java-sdk-s3

1.12.638

com.amazonaws

aws-java-sdk-ses

1.12.638

com.amazonaws

aws-java-sdk-simpledb

1.12.638

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.638

com.amazonaws

aws-java-sdk-sns

1.12.638

com.amazonaws

aws-java-sdk-sqs

1.12.638

com.amazonaws

aws-java-sdk-ssm

1.12.638

com.amazonaws

aws-java-sdk-storagegateway

1.12.638

com.amazonaws

aws-java-sdk-sts

1.12.638

com.amazonaws

aws-java-sdk-support

1.12.638

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.638

com.amazonaws

jmespath-java

1.12.638

com.azure

azure-core

1.30.0

com.azure

azure-core-http-netty

1.12.3

com.azure

azure-identity

1.5.3

com.azure

azure-security-keyvault-keys

4.4.4

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.27.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.15.2

com.fasterxml.jackson.core

jackson-core

2.15.2

com.fasterxml.jackson.core

jackson-databind

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-xml

2.13.3

com.fasterxml.jackson.dataformat

jackson-dataformat-yaml

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.16.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.15.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.15.2

com.fasterxml.woodstox

woodstox-core

6.4.0

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.5-4

com.github.stephenc.jcip

jcip-annotations

1.0-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.10.1

com.google.crypto.tink

tink

1.9.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

23.5.26

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

3.25.1

com.helger

profiler

1.1.1

com.ibm.icu

icu4j

75.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.azure

msal4j

1.13.0

com.microsoft.azure

msal4j-persistence-extension

1.1.0

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.microsoft.sqlserver

mssql-jdbc

11.2.3.jre8

com.nimbusds

content-type

2.2

com.nimbusds

lang-tag

1.6

com.nimbusds

nimbus-jose-jwt

9.31

com.nimbusds

oauth2-oidc-sdk

9.35

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.4.3

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.16.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.13.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.27

io.delta

delta-sharing-client_2.12

1.2.4

io.dropwizard.metrics

metrics-annotation

4.2.19

io.dropwizard.metrics

metrics-core

4.2.19

io.dropwizard.metrics

metrics-graphite

4.2.19

io.dropwizard.metrics

metrics-healthchecks

4.2.19

io.dropwizard.metrics

metrics-jetty9

4.2.19

io.dropwizard.metrics

metrics-jmx

4.2.19

io.dropwizard.metrics

metrics-json

4.2.19

io.dropwizard.metrics

metrics-jvm

4.2.19

io.dropwizard.metrics

metrics-servlets

4.2.19

io.netty

netty-all

4.1.108.Final

io.netty

netty-buffer

4.1.108.Final

io.netty

netty-codec

4.1.108.Final

io.netty

netty-codec-dns

4.1.77.Final

io.netty

netty-codec-http

4.1.108.Final

io.netty

netty-codec-http2

4.1.108.Final

io.netty

netty-codec-socks

4.1.108.Final

io.netty

netty-common

4.1.108.Final

io.netty

netty-handler

4.1.108.Final

io.netty

netty-handler-proxy

4.1.108.Final

io.netty

netty-resolver

4.1.108.Final

io.netty

netty-resolver-dns

4.1.77.Final

io.netty

netty-resolver-dns-classes-macos

4.1.77.Final

io.netty

netty-resolver-dns-native-macos

4.1.77.Final-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-db-r16-windows-x86_64

io.netty

netty-tcnative-classes

2.0.61.Final

io.netty

netty-transport

4.1.108.Final

io.netty

netty-transport-classes-epoll

4.1.108.Final

io.netty

netty-transport-classes-kqueue

4.1.108.Final

io.netty

netty-transport-native-epoll

4.1.108.Final

io.netty

netty-transport-native-epoll

4.1.108.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.108.Final-linux-riscv64

io.netty

netty-transport-native-epoll

4.1.108.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.108.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.108.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.108.Final

io.projectreactor

reactor-core

3.4.19

io.projectreactor.netty

reactor-netty-core

1.0.20

io.projectreactor.netty

reactor-netty-http

1.0.20

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

net.java.dev.jna

jna

5.8.0

net.java.dev.jna

jna-platform

5.6.0

net.minidev

accessors-smart

2.4.11

net.minidev

json-smart

2.4.11

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.10.11

org.apache.ant

ant-jsch

1.10.11

org.apache.ant

ant-launcher

1.10.11

org.apache.arrow

arrow-format

15.0.0

org.apache.arrow

arrow-memory-core

15.0.0

org.apache.arrow

arrow-memory-netty

15.0.0

org.apache.arrow

arrow-vector

15.0.0

org.apache.avro

avro

1.11.3

org.apache.avro

avro-ipc

1.11.3

org.apache.avro

avro-mapred

1.11.3

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.23.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.6

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.2

org.apache.logging.log4j

log4j-1.2-api

2.22.1

org.apache.logging.log4j

log4j-api

2.22.1

org.apache.logging.log4j

log4j-core

2.22.1

org.apache.logging.log4j

log4j-layout-template-json

2.22.1

org.apache.logging.log4j

log4j-slf4j2-impl

2.22.1

org.apache.orc

orc-core

1.9.2-shaded-protobuf

org.apache.orc

orc-mapreduce

1.9.2-shaded-protobuf

org.apache.orc

orc-shims

1.9.2

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.ws.xmlschema

xmlschema-core

2.3.0

org.apache.xbean

xbean-asm9-shaded

4.23

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.9.2

org.apache.zookeeper

zookeeper-jute

3.9.2

org.bouncycastle

bcprov-jdk15on

1.70

org.checkerframework

checker-qual

3.31.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.codehaus.woodstox

stax2-api

4.2.1

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.collections

eclipse-collections

11.1.0

org.eclipse.collections

eclipse-collections-api

11.1.0

org.eclipse.jetty

jetty-client

9.4.52.v20230823

org.eclipse.jetty

jetty-continuation

9.4.52.v20230823

org.eclipse.jetty

jetty-http

9.4.52.v20230823

org.eclipse.jetty

jetty-io

9.4.52.v20230823

org.eclipse.jetty

jetty-jndi

9.4.52.v20230823

org.eclipse.jetty

jetty-plus

9.4.52.v20230823

org.eclipse.jetty

jetty-proxy

9.4.52.v20230823

org.eclipse.jetty

jetty-security

9.4.52.v20230823

org.eclipse.jetty

jetty-server

9.4.52.v20230823

org.eclipse.jetty

jetty-servlet

9.4.52.v20230823

org.eclipse.jetty

jetty-servlets

9.4.52.v20230823

org.eclipse.jetty

jetty-util

9.4.52.v20230823

org.eclipse.jetty

jetty-util-ajax

9.4.52.v20230823

org.eclipse.jetty

jetty-webapp

9.4.52.v20230823

org.eclipse.jetty

jetty-xml

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-api

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-client

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-common

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-server

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-servlet

9.4.52.v20230823

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.40

org.glassfish.jersey.containers

jersey-container-servlet-core

2.40

org.glassfish.jersey.core

jersey-client

2.40

org.glassfish.jersey.core

jersey-common

2.40

org.glassfish.jersey.core

jersey-server

2.40

org.glassfish.jersey.inject

jersey-hk2

2.40

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0-databricks-1

org.mlflow

mlflow-spark_2.12

2.9.1

org.objenesis

objenesis

2.5.1

org.ow2.asm

asm

9.3

org.postgresql

postgresql

42.6.1

org.reactivestreams

reactive-streams

1.0.4

org.roaringbitmap

RoaringBitmap

0.9.45-databricks

org.roaringbitmap

shims

0.9.45-databricks

org.rocksdb

rocksdbjni

9.2.1

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.11.0

org.scala-lang.modules

scala-java8-compat_2.12

0.9.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.16

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.16

org.scalatest

scalatest-core_2.12

3.2.16

org.scalatest

scalatest-diagrams_2.12

3.2.16

org.scalatest

scalatest-featurespec_2.12

3.2.16

org.scalatest

scalatest-flatspec_2.12

3.2.16

org.scalatest

scalatest-freespec_2.12

3.2.16

org.scalatest

scalatest-funspec_2.12

3.2.16

org.scalatest

scalatest-funsuite_2.12

3.2.16

org.scalatest

scalatest-matchers-core_2.12

3.2.16

org.scalatest

scalatest-mustmatchers_2.12

3.2.16

org.scalatest

scalatest-propspec_2.12

3.2.16

org.scalatest

scalatest-refspec_2.12

3.2.16

org.scalatest

scalatest-shouldmatchers_2.12

3.2.16

org.scalatest

scalatest-wordspec_2.12

3.2.16

org.scalatest

scalatest_2.12

3.2.16

org.slf4j

jcl-over-slf4j

2.0.7

org.slf4j

jul-to-slf4j

2.0.7

org.slf4j

slf4j-api

2.0.7

org.slf4j

slf4j-simple

1.7.25

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

2.4.1-linux-x86_64

stax

stax-api

1.0.1

Última atualização em