Databricks Runtime 10.4 LTS

The following release notes provide information about Databricks Runtime 10.4 LTS and Databricks Runtime 10.4 LTS Photon, powered by Apache Spark 3.2.1. Photon is in Public Preview. Databricks released this version in March 2022.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements

Iceberg to Delta table converter (Public Preview)

Convert to Delta now supports converting an Iceberg table to a Delta table in place. It does this by using Iceberg native metadata and file manifests. See Convert to Delta Lake.

Auto Compaction rollbacks are now enabled by default

This release improves the behavior for Delta Lake writes that commit when there are concurrent Auto Compaction transactions. Before this release, such writes would often quit, due to concurrent modifications to a table. Writes will now succeed even if there are concurrent Auto Compaction transactions.

Low Shuffle Merge is now enabled by default

The MERGE INTO command now always uses the new low-shuffle implementation. This behavior improves the performance of the MERGE INTO command significantly for most workloads. The configuration setting that was previously used to enable this feature has been removed. See Low shuffle merge on Databricks.

Insertion order tags are now preserved for UPDATEs and DELETEs

The UPDATE and DELETE commands now preserve existing clustering information (including Z-ordering) for files that are updated or deleted. This behavior is a best-effort approach, and this approach does not apply to cases when files are so small that these files are combined during the update or delete.

HikariCP is now the default Hive metastore connection pool

HikariCP brings many stability improvements for Hive metastore access while maintaining fewer connections compared to the previous BoneCP connection pool implementation. HikariCP is enabled by default on any Databricks Runtime cluster that uses the Databricks Hive metastore (for example, when spark.sql.hive.metastore.jars is not set). You can also explicitly switch to other connection pool implementations, for example BoneCP, by setting spark.databricks.hive.metastore.client.pool.type.

Azure Synapse connector now enables the maximum number of allowed reject rows to be set

The Azure Synapse connector now supports a maxErrors DataFrame option. This update enables you to configure the maximum number of rejected rows that are allowed during reads and writes before the load operation is cancelled. All rejected rows are ignored. For example, if two out of ten records have errors, only eight records are processed.

This option maps directly to the REJECT_VALUE option for the CREATE EXTERNAL TABLE statement in PolyBase and to the MAXERRORS option for the Azure Synapse connector’s COPY command.

By default, maxErrors value is set to 0: all records are expected to be valid.

Asynchronous state checkpointing is now generally available

You can enable asynchronous state checkpointing in stateful streaming queries with large state updates. This can reduce the end-to-end micro-batch latency. This feature is now generally available. See Asynchronous state checkpointing for stateful queries.

Parameter defaults can now be specified for SQL user-defined functions

When you create a SQL user-defined function (SQL UDF), you can now specify default expressions for the SQL UDF’s parameters. You can then call the SQL UDF without providing arguments for those parameters, and Databricks will fill in the default values for those parameters. See CREATE FUNCTION (SQL and Python).

New Spark SQL functions

The following Spark SQL functions are now available with this release:

  • try_multiply: Returns multiplier multiplied by multiplicand, or NULL on overflow.

  • try_subtract: Returns the subtraction of expr2 from expr1, or NULL on overflow.

New working directory for High Concurrency clusters

On High Concurrency clusters with either table access control or credential passthrough enabled, the current working directory of notebooks is now the user’s home directory. Previously, the working directory was /databricks/driver.

Identity columns support in Delta tables is now generally available

Delta Lake now supports identity columns. When you write to a Delta table that defines an identity column, and you do not provide values for that column, Delta now automatically assigns a unique and statistically increasing or decreasing value. See CREATE TABLE [USING].

Library upgrades

  • Upgraded Python libraries:

    • filelock from 3.4.2 to 3.6.0

  • Upgraded R libraries:

    • brew from 1.0-6 to 1.0-7

    • broom from 0.7.11 to 0.7.12

    • cli from 3.1.0 to 3.2.0

    • clipr from 0.7.1 to 0.8.0

    • colorspace from 2.0-2 to 2.0-3

    • crayon from 1.4.2 to 1.5.0

    • dplyr from 1.0.7 to 1.0.8

    • dtplyr from 1.2.0 to 1.2.1

    • evaluate from 0.14 to 0.15

    • foreach from 1.5.1 to 1.5.2

    • future from 1.23.0 to 1.24.0

    • generics from 0.1.1 to 0.1.2

    • glue from 1.6.0 to 1.6.1

    • gower from 0.2.2 to 1.0.0

    • iterators from 1.0.13 to 1.0.14

    • jsonlite from 1.7.3 to 1.8.0

    • magrittr from 2.0.1 to 2.0.2

    • mgcv from 1.8-38 to 1.8-39

    • pillar from 1.6.4 to 1.7.0

    • randomForest from 4.6-14 to 4.7-1

    • readr from 2.1.1 to 2.1.2

    • recipes from 0.1.17 to 0.2.0

    • rlang from 0.4.12 to 1.0.1

    • rpart from 4.1-15 to 4.1.16

    • RSQLite from 2.2.9 to 2.2.10

    • sparklyr from 1.7.4 to 1.7.5

    • testthat from 3.1.1 to 3.1.2

    • tidyr from 1.1.4 to 1.2.0

    • tidyselect from 1.1.1 to 1.1.2

    • tinytex from 0.36 to 0.37

    • yaml from 2.2.1 to 2.3.5

  • Upgraded Java libraries:

    • io.delta.delta-sharing-spark_2.12 from 0.3.0 to 0.4.0

Apache Spark

Databricks Runtime 10.4 LTS includes Apache Spark 3.2.1. This release includes all Spark fixes and improvements included in Databricks Runtime 10.3 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-38322] [SQL] Support query stage show runtime statistics in formatted explain mode

  • [SPARK-38162] [SQL] Optimize one row plan in normal and AQE Optimizer

  • [SPARK-38229] [SQL] Should’t check temp/external/ifNotExists with visitReplaceTable when parser

  • [SPARK-34183] [SS] DataSource V2: Required distribution and ordering in micro-batch execution

  • [SPARK-37932] [SQL]Wait to resolve missing attributes before applying DeduplicateRelations

  • [SPARK-37904] [SQL] Improve RebalancePartitions in rules of Optimizer

  • [SPARK-38236] [SQL][3.2][3.1] Check if table location is absolute by “new Path(locationUri).isAbsolute” in create/alter table

  • [SPARK-38035] [SQL] Add docker tests for build-in JDBC dialect

  • [SPARK-38042] [SQL] Ensure that ScalaReflection.dataTypeFor works on aliased array types

  • [SPARK-38273] [SQL] decodeUnsafeRows’s iterators should close underlying input streams

  • [SPARK-38311] [SQL] Fix DynamicPartitionPruning/BucketedReadSuite/ExpressionInfoSuite under ANSI mode

  • [SPARK-38305] [CORE] Explicitly check if source exists in unpack() before calling FileUtil methods

  • [SPARK-38275] [SS] Include the writeBatch’s memory usage as the total memory usage of RocksDB state store

  • [SPARK-38132] [SQL] Remove NotPropagation rule

  • [SPARK-38286] [SQL] Union’s maxRows and maxRowsPerPartition may overflow

  • [SPARK-38306] [SQL] Fix ExplainSuite,StatisticsCollectionSuite and StringFunctionsSuite under ANSI mode

  • [SPARK-38281] [SQL][Tests] Fix AnalysisSuite under ANSI mode

  • [SPARK-38307] [SQL][Tests] Fix ExpressionTypeCheckingSuite and CollectionExpressionsSuite under ANSI mode

  • [SPARK-38300] [SQL] Use ByteStreams.toByteArray to simplify fileToString and resourceToBytes in catalyst.util

  • [SPARK-38304] [SQL] Elt() should return null if index is null under ANSI mode

  • [SPARK-38271] PoissonSampler may output more rows than MaxRows

  • [SPARK-38297] [PYTHON] Explicitly cast the return value at DataFrame.to_numpy in POS

  • [SPARK-38295] [SQL][Tests] Fix ArithmeticExpressionSuite under ANSI mode

  • [SPARK-38290] [SQL] Fix JsonSuite and ParquetIOSuite under ANSI mode

  • [SPARK-38299] [SQL] Clean up deprecated usage of StringBuilder.newBuilder

  • [SPARK-38060] [SQL] Respect allowNonNumericNumbers when parsing quoted NaN and Infinity values in JSON reader

  • [SPARK-38276] [SQL] Add approved TPCDS plans under ANSI mode

  • [SPARK-38206] [SS] Ignore nullability on comparing the data type of join keys on stream-stream join

  • [SPARK-37290] [SQL] - Exponential planning time in case of non-deterministic function

  • [SPARK-38232] [SQL] Explain formatted does not collect subqueries under query stage in AQE

  • [SPARK-38283] [SQL] Test invalid datetime parsing under ANSI mode

  • [SPARK-38140] [SQL] Desc column stats (min, max) for timestamp type is not consistent with the values due to time zone difference

  • [SPARK-38227] [SQL][SS] Apply strict nullability of nested column in time window / session window

  • [SPARK-38221] [SQL] Eagerly iterate over groupingExpressions when moving complex grouping expressions out of an Aggregate node

  • [SPARK-38216] [SQL] Fail early if all the columns are partitioned columns when creating a Hive table

  • [SPARK-38214] [SS]No need to filter windows when windowDuration is multiple of slideDuration

  • [SPARK-38182] [SQL] Fix NoSuchElementException if pushed filter does not contain any references

  • [SPARK-38159] [SQL] Add a new FileSourceMetadataAttribute for the Hidden File Metadata

  • [SPARK-38123] [SQL] Unified use DataType as targetType of QueryExecutionErrors#castingCauseOverflowError

  • [SPARK-38118] [SQL] Func(wrong data type) in HAVING clause should throw data mismatch error

  • [SPARK-35173] [SQL][PYTHON] Add multiple columns adding support

  • [SPARK-38177] [SQL] Fix wrong transformExpressions in Optimizer

  • [SPARK-38228] [SQL] Legacy store assignment should not fail on error under ANSI mode

  • [SPARK-38173] [SQL] Quoted column cannot be recognized correctly when quotedRegexColumnNa…

  • [SPARK-38130] [SQL] Remove array_sort orderable entries check

  • [SPARK-38199] [SQL] Delete the unused dataType specified in the definition of IntervalColumnAccessor

  • [SPARK-38203] [SQL] Fix SQLInsertTestSuite and SchemaPruningSuite under ANSI mode

  • [SPARK-38163] [SQL] Preserve the error class of SparkThrowable while constructing of function builder

  • [SPARK-38157] [SQL] Explicitly set ANSI to false in test timestampNTZ/timestamp.sql and SQLQueryTestSuite to match the expected golden results

  • [SPARK-38069] [SQL][SS] Improve the calculation of time window

  • [SPARK-38164] [SQL] New SQL functions: try_subtract and try_multiply

  • [SPARK-38176] [SQL] ANSI mode: allow implicitly casting String to other simple types

  • [SPARK-37498] [PYTHON] Add eventually for test_reuse_worker_of_parallelize_range

  • [SPARK-38198] [SQL][3.2] Fix QueryExecution.debug#toFile use the passed in maxFields when explainMode is CodegenMode

  • [SPARK-38131] [SQL] Use error classes in user-facing exceptions only

  • [SPARK-37652] [SQL] Add test for optimize skewed join through union

  • [SPARK-37585] [SQL] Update InputMetric in DataSourceRDD with TaskCompletionListener

  • [SPARK-38113] [SQL] Use error classes in the execution errors of pivoting

  • [SPARK-38178] [SS] Correct the logic to measure the memory usage of RocksDB

  • [SPARK-37969] [SQL] HiveFileFormat should check field name

  • [SPARK-37652] Revert “[SQL]Add test for optimize skewed join through union”

  • [SPARK-38124] [SQL][SS] Introduce StatefulOpClusteredDistribution and apply to stream-stream join

  • [SPARK-38030] [SQL] Canonicalization should not remove nullability of AttributeReference dataType

  • [SPARK-37907] [SQL] InvokeLike support ConstantFolding

  • [SPARK-37891] [CORE] Add scalastyle check to disable scala.concurrent.ExecutionContext.Implicits.global

  • [SPARK-38150] [SQL] Update comment of RelationConversions

  • [SPARK-37943] [SQL] Use error classes in the compilation errors of grouping

  • [SPARK-37652] [SQL]Add test for optimize skewed join through union

  • [SPARK-38056] [Web UI][3.2] Fix issue of Structured streaming not working in history server when using LevelDB

  • [SPARK-38144] [CORE] Remove unused spark.storage.safetyFraction config

  • [SPARK-38120] [SQL] Fix HiveExternalCatalog.listPartitions when partition column name is upper case and dot in partition value

  • [SPARK-38122] [Docs] Update the App Key of DocSearch

  • [SPARK-37479] [SQL] Migrate DROP NAMESPACE to use V2 command by default

  • [SPARK-35703] [SQL] Relax constraint for bucket join and remove HashClusteredDistribution

  • [SPARK-37983] [SQL] Back out agg build time metrics from sort aggregate

  • [SPARK-37915] [SQL] Combine unions if there is a project between them

  • [SPARK-38105] [SQL] Use error classes in the parsing errors of joins

  • [SPARK-38073] [PYTHON] Update atexit function to avoid issues with late binding

  • [SPARK-37941] [SQL] Use error classes in the compilation errors of casting

  • [SPARK-37937] [SQL] Use error classes in the parsing errors of lateral join

  • [SPARK-38100] [SQL] Remove unused private method in Decimal

  • [SPARK-37987] [SS] Fix flaky test StreamingAggregationSuite.changing schema of state when restarting query

  • [SPARK-38003] [SQL] LookupFunctions rule should only look up functions from the scalar function registry

  • [SPARK-38075] [SQL] Fix hasNext in HiveScriptTransformationExec’s process output iterator

  • [SPARK-37965] [SQL] Remove check field name when reading/writing existing data in Orc

  • [SPARK-37922] [SQL] Combine to one cast if we can safely up-cast two casts (for dbr-branch-10.x)

  • [SPARK-37675] [SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized

  • [SPARK-38011] [SQL] Remove duplicated and useless configuration in ParquetFileFormat

  • [SPARK-37929] [SQL] Support cascade mode for dropNamespace API

  • [SPARK-37931] [SQL] Quote the column name if needed

  • [SPARK-37990] [SQL] Support TimestampNTZ in RowToColumnConverter

  • [SPARK-38001] [SQL] Replace the error classes related to unsupported features by UNSUPPORTED_FEATURE

  • [SPARK-37839] [SQL] DS V2 supports partial aggregate push-down AVG

  • [SPARK-37878] [SQL] Migrate SHOW CREATE TABLE to use v2 command by default

  • [SPARK-37731] [SQL] Refactor and cleanup function lookup in Analyzer

  • [SPARK-37979] [SQL] Switch to more generic error classes in AES functions

  • [SPARK-37867] [SQL] Compile aggregate functions of build-in JDBC dialect

  • [SPARK-38028] [SQL] Expose Arrow Vector from ArrowColumnVector

  • [SPARK-30062] [SQL] Add the IMMEDIATE statement to the DB2 dialect truncate implementation

  • [SPARK-36649] [SQL] Support Trigger.AvailableNow on Kafka data source

  • [SPARK-38018] [SQL] Fix ColumnVectorUtils.populate to handle CalendarIntervalType correctly

  • [SPARK-38023] [CORE] ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished

  • [SPARK-38019] [CORE] Make ExecutorMonitor.timedOutExecutors deterministic

  • [SPARK-37957] [SQL] Correctly pass deterministic flag for V2 scalar functions

  • [SPARK-37985] [SQL] Fix flaky test for SPARK-37578

  • [SPARK-37986] [SQL] Support TimestampNTZ in radix sort

  • [SPARK-37967] [SQL] Literal.create support ObjectType

  • [SPARK-37827] [SQL] Put the some built-in table properties into V1Table.propertie to adapt to V2 command

  • [SPARK-37963] [SQL] Need to update Partition URI after renaming table in InMemoryCatalog

  • [SPARK-35442] [SQL] Support propagate empty relation through aggregate/union

  • [SPARK-37933] [SQL] Change the traversal method of V2ScanRelationPushDown push down rules

  • [SPARK-37917] [SQL] Push down limit 1 for right side of left semi/anti join if join condition is empty

  • [SPARK-37959] [ML] Fix the UT of checking norm in KMeans & BiKMeans

  • [SPARK-37906] [SQL] spark-sql should not pass last comment to backend

  • [SPARK-37627] [SQL] Add sorted column in BucketTransform

System environment

  • Operating System: Ubuntu 20.04.4 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.14

  • Python: 3.8.10

  • R: 4.1.2

  • Delta Lake: 1.1.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

appdirs

1.4.4

argon2-cffi

20.1.0

async-generator

1.10

attrs

20.3.0

backcall

0.2.0

bidict

0.21.4

bleach

3.3.0

boto3

1.16.7

botocore

1.19.7

certifi

2020.12.5

cffi

1.14.5

chardet

4.0.0

cycler

0.10.0

Cython

0.29.23

dbus-python

1.2.16

decorator

5.0.6

defusedxml

0.7.1

distlib

0.3.4

distro-info

0.23ubuntu1

entrypoints

0.3

facets-overview

1.0.0

filelock

3.6.0

idna

2.10

ipykernel

5.3.4

ipython

7.22.0

ipython-genutils

0.2.0

ipywidgets

7.6.3

jedi

0.17.2

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.0.1

jsonschema

3.2.0

jupyter-client

6.1.12

jupyter-core

4.7.1

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.1

koalas

1.8.2

MarkupSafe

2.0.1

matplotlib

3.4.2

mistune

0.8.4

nbclient

0.5.3

nbconvert

6.0.7

nbformat

5.1.3

nest-asyncio

1.5.1

notebook

6.3.0

numpy

1.20.1

packaging

20.9

pandas

1.2.4

pandocfilters

1.4.3

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.2.0

pip

21.0.1

plotly

5.5.0

prometheus-client

0.10.1

prompt-toolkit

3.0.17

protobuf

3.17.2

psycopg2

2.8.5

ptyprocess

0.7.0

pyarrow

4.0.0

pycparser

2.20

Pygments

2.8.1

PyGObject

3.36.0

pyparsing

2.4.7

pyrsistent

0.17.3

python-apt

2.0.0+ubuntu0.20.4.7

python-dateutil

2.8.1

python-engineio

4.3.0

python-socketio

5.4.1

pytz

2020.5

pyzmq

20.0.0

requests

2.25.1

requests-unixsocket

0.2.0

s3transfer

0.3.7

scikit-learn

0.24.1

scipy

1.6.2

seaborn

0.11.1

Send2Trash

1.5.0

setuptools

52.0.0

six

1.15.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

terminado

0.9.4

testpath

0.4.4

threadpoolctl

2.1.0

tornado

6.1

traitlets

5.0.5

unattended-upgrades

0.1

urllib3

1.25.11

virtualenv

20.4.1

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.36.2

widgetsnbextension

3.5.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-02-24.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.1.2

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.2

boot

1.3-28

brew

1.0-7

brio

1.1.3

broom

0.7.12

bslib

0.3.1

cachem

1.0.6

callr

3.7.0

caret

6.0-90

cellranger

1.1.0

chron

2.3-56

class

7.3-20

cli

3.2.0

clipr

0.8.0

cluster

2.1.2

codetools

0.2-18

colorspace

2.0-3

commonmark

1.7

compiler

4.1.2

config

0.3.1

cpp11

0.4.2

crayon

1.5.0

credentials

1.3.2

curl

4.3.2

data.table

1.14.2

datasets

4.1.2

DBI

1.1.2

dbplyr

2.1.1

desc

1.4.0

devtools

2.4.3

diffobj

0.3.5

digest

0.6.29

dplyr

1.0.8

dtplyr

1.2.1

e1071

1.7-9

ellipsis

0.3.2

evaluate

0.15

fansi

1.0.2

farver

2.1.0

fastmap

1.1.0

fontawesome

0.2.2

forcats

0.5.1

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.24.0

future.apply

1.8.1

gargle

1.2.0

generics

0.1.2

gert

1.5.0

ggplot2

3.3.5

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-3

globals

0.14.0

glue

1.6.1

googledrive

2.0.0

googlesheets4

1.0.0

gower

1.0.0

graphics

4.1.2

grDevices

4.1.2

grid

4.1.2

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

hardhat

0.2.0

haven

2.4.3

highr

0.9

hms

1.1.1

htmltools

0.5.2

htmlwidgets

1.5.4

httpuv

1.6.5

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ids

1.0.1

ini

0.3.1

ipred

0.9-12

isoband

0.2.5

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.0

KernSmooth

2.23-20

knitr

1.37

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.6.10

lifecycle

1.0.1

listenv

0.8.0

lubridate

1.8.0

magrittr

2.0.2

markdown

1.1

MASS

7.3-55

Matrix

1.4-0

memoise

2.0.1

methods

4.1.2

mgcv

1.8-39

mime

0.12

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-155

nnet

7.3-17

numDeriv

2016.8-1.1

openssl

1.4.6

parallel

4.1.2

parallelly

1.30.0

pillar

1.7.0

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgload

1.2.4

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.5.2

prodlim

2019.11.13

progress

1.2.2

progressr

0.10.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-26

ps

1.6.0

purrr

0.3.4

r2d3

0.2.5

R6

2.5.1

randomForest

4.7-1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-2

Rcpp

1.0.8

RcppEigen

0.3.3.9.1

readr

2.1.2

readxl

1.3.1

recipes

0.2.0

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.1

reshape2

1.4.4

rlang

1.0.1

rmarkdown

2.11

RODBC

1.3-19

roxygen2

7.1.2

rpart

4.1.16

rprojroot

2.0.2

Rserve

1.8-10

RSQLite

2.2.10

rstudioapi

0.13

rversions

2.1.1

rvest

1.0.2

sass

0.4.0

scales

1.1.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.1

sourcetools

0.1.7

sparklyr

1.7.5

SparkR

3.2.0

spatial

7.3-11

splines

4.1.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.2

stats4

4.1.2

stringi

1.7.6

stringr

1.4.0

survival

3.2-13

sys

3.4

tcltk

4.1.2

TeachingDemos

2.10

testthat

3.1.2

tibble

3.1.6

tidyr

1.2.0

tidyselect

1.1.2

tidyverse

1.3.1

timeDate

3043.102

tinytex

0.37

tools

4.1.2

tzdb

0.2.0

usethis

2.1.5

utf8

1.2.2

utils

4.1.2

uuid

1.0-3

vctrs

0.3.8

viridisLite

0.4.0

vroom

1.5.7

waldo

0.3.1

whisker

0.4

withr

2.4.3

xfun

0.29

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.5

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-glue

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.12.3

com.fasterxml.jackson.core

jackson-core

2.12.3

com.fasterxml.jackson.core

jackson-databind

2.12.3

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.12.3

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.12.3

com.fasterxml.jackson.module

jackson-module-paranamer

2.12.3

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.12.3

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.0-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.0

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

9.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.istack

istack-commons-runtime

3.0.8

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.8.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

hive-2.3__hadoop-3.2

jets3t-0.7

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.4.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.68.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.10

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

maven-trees

hive-2.3__hadoop-3.2

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.3

net.snowflake

spark-snowflake_2.12

2.9.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.10.2

org.apache.avro

avro-ipc

1.10.2

org.apache.avro

avro-mapred

1.10.2

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.1-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.1

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.htrace

htrace-core4

4.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.5.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.6.12

org.apache.orc

orc-mapreduce

1.6.12

org.apache.orc

orc-shims

1.6.12

org.apache.parquet

parquet-column

1.12.0-databricks-0003

org.apache.parquet

parquet-common

1.12.0-databricks-0003

org.apache.parquet

parquet-encoding

1.12.0-databricks-0003

org.apache.parquet

parquet-format-structures

1.12.0-databricks-0003

org.apache.parquet

parquet-hadoop

1.12.0-databricks-0003

org.apache.parquet

parquet-jackson

1.12.0-databricks-0003

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.43.v20210629

org.eclipse.jetty

jetty-continuation

9.4.43.v20210629

org.eclipse.jetty

jetty-http

9.4.43.v20210629

org.eclipse.jetty

jetty-io

9.4.43.v20210629

org.eclipse.jetty

jetty-jndi

9.4.43.v20210629

org.eclipse.jetty

jetty-plus

9.4.43.v20210629

org.eclipse.jetty

jetty-proxy

9.4.43.v20210629

org.eclipse.jetty

jetty-security

9.4.43.v20210629

org.eclipse.jetty

jetty-server

9.4.43.v20210629

org.eclipse.jetty

jetty-servlet

9.4.43.v20210629

org.eclipse.jetty

jetty-servlets

9.4.43.v20210629

org.eclipse.jetty

jetty-util

9.4.43.v20210629

org.eclipse.jetty

jetty-util-ajax

9.4.43.v20210629

org.eclipse.jetty

jetty-webapp

9.4.43.v20210629

org.eclipse.jetty

jetty-xml

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-api

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-client

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-common

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-server

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-servlet

9.4.43.v20210629

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jaxb

jaxb-runtime

2.3.2

org.glassfish.jersey.containers

jersey-container-servlet

2.34

org.glassfish.jersey.containers

jersey-container-servlet-core

2.34

org.glassfish.jersey.core

jersey-client

2.34

org.glassfish.jersey.core

jersey-common

2.34

org.glassfish.jersey.core

jersey-server

2.34

org.glassfish.jersey.inject

jersey-hk2

2.34

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.2.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.2.19

org.roaringbitmap

RoaringBitmap

0.9.14

org.roaringbitmap

shims

0.9.14

org.rocksdb

rocksdbjni

6.20.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1