Databricks Runtime 11.3 LTS

The following release notes provide information about Databricks Runtime 11.3 LTS, powered by Apache Spark 3.3.0. Databricks released these images in October 2022.

New features and improvements

Databricks Kinesis connector now supports reading from Kinesis Data streams in EFO mode

You can now use the Databricks Kinesis structured streaming source in Databricks Runtime 11.3 LTS to run queries that read from Kinesis Data streams in enhanced fan-out mode. This allows dedicated throughput per shard, per consumer and record delivery in push mode.

New H3 geospatial functions and added Photon support for all H3 functions

Introducing 4 new H3 functions, h3_maxchild, h3_minchild, h3_pointash3, and h3_pointash3string. These functions are available in SQL, Scala, and Python. All H3 expressions are now supported in Photon. See H3 geospatial functions.

Increasing initial partitions to scan for selective queries

The value of initial partitions to scan has been increased to 10 for selective query with take/tail/limit in Photon-enabled clusters and LIMIT in Databricks SQL. With 10 partitions, you can avoid the overhead of launching multiple small jobs and a slow scale-up. You can also configure this through spark.sql.limit.selectiveInitialNumPartitions.

New AQE plan versions visualization

Introducing AQE plan versions that allows you to visualize your runtime plan updates from adaptive query execution (AQE).

New asynchronous progress tracking and log purging modes

Introducing Structured Streaming modes called asynchronous progress tracking and asynchronous log purging. Asynchronous log purging mode lowers the latency of streaming queries by removing logs used for progress tracking in the background.

Structured Streaming on Unity Catalog now supports display()

You can now use display() when you use Structured Streaming to work with tables registered in Unity Catalog.

Pipeline events are now logged in JSON format

Databricks now writes pipeline events to the driver log in JSON format. While each event will be JSON-parseable, large events may not contain all fields, or the fields may be truncated. Each event is logged in a single line with the prefix Event received: . The following is an example event.

Event received: {"id":"some-event-id","origin":{"pipeline_id":"some-pipeline-id","cluster_id":"some-cluster id"},"message":"simple [truncated] message","level":"WARN"}

Arbitrary Stateful Processing in Structured Streaming with Python

Introducing the applyInPandasWithState function that can be used to perform arbitrary stateful processing in PySpark. This is equivalent to the flatMapGroupsWithState function in the Java API.

Date inference in CSV files

Introducing improved inference of date type columns in CSV files. When the date format is consistent across the records for a column, those columns can be inferred as DateType. You can also have a combination of date formats across different columns. Databricks can automatically infer the date format for each column. Date columns in CSV files prior to Databricks Runtime 11.3 LTS are left as StringType.

Clone support for Apache Parquet and Apache Iceberg tables (Public Preview)

Clone can now be used to create and incrementally update Delta tables that mirror Apache Parquet and Apache Iceberg tables. You can update your source Parquet table and incrementally apply the changes to their cloned Delta table with the clone command. See Incrementally clone Parquet and Iceberg tables to Delta Lake.

Behavior changes

Updated Databricks Snowflake connector

The Databricks Snowflake connector has been updated to the latest version of code from the open-source repository, Snowflake Data Source for Apache Spark. It is now fully compatible with Databricks Runtime 11.3 LTS, including predicate pushdown and internal query plan pushdown while maintaining all of the features of the open-source version.

Hadoop cache for S3A is now disabled

The Hadoop cache (FileSystem Apache Hadoop Main 3.3.4 API) for S3A is now disabled. This is to align with other cloud storage connectors. For workloads that rely on file system caching, make sure that newly created file systems are supplied with the correct Hadoop configurations, including credential providers.

Delta Lake stats collection schema now matches column order in table schema definition

This change addresses a bug in the Delta Lake protocol where stats were not collected for columns due to a mismatch in DataFrame and table column ordering. In some cases, you might encounter write performance degradation due to stats collection on previously untracked fields. See Data skipping with Z-order indexes for Delta Lake.

applyInPandasWithState throws an error if the query has a shuffle after the operator

The operator applyInPandasWithState throws an error if the query has shuffle after the operator. This happens when either the user adds shuffle after the operation, or the optimizer or sink adds shuffle implicitly.

Library upgrades

  • Upgraded Python libraries:

    • distlib from 0.3.5 to 0.3.6

  • Upgraded R libraries:

    • broom from 1.0.0 to 1.0.1

    • callr from 3.7.1 to 3.7.2

    • dplyr from 1.0.9 to 1.0.10

    • dtplyr from 1.2.1 to 1.2.2

    • forcats from 0.5.1 to 0.5.2

    • future from 1.27.0 to 1.28.0

    • future.apply from 1.9.0 to 1.9.1

    • gert from 1.7.0 to 1.8.0

    • globals from 0.16.0 to 0.16.1

    • gtable from 0.3.0 to 0.3.1

    • haven from 2.5.0 to 2.5.1

    • hms from 1.1.1 to 1.1.2

    • httr from 1.4.3 to 1.4.4

    • knitr from 1.39 to 1.40

    • modelr from 0.1.8 to 0.1.9

    • pillar from 1.8.0 to 1.8.1

    • progressr from 0.10.1 to 0.11.0

    • readxl from 1.4.0 to 1.4.1

    • reprex from 2.0.1 to 2.0.2

    • rlang from 1.0.4 to 1.0.5

    • rmarkdown from 2.14 to 2.16

    • RSQLite from 2.2.15 to 2.2.16

    • rstudioapi from 0.13 to 0.14

    • rversions from 2.1.1 to 2.1.2

    • rvest from 1.0.2 to 1.0.3

    • scales from 1.2.0 to 1.2.1

    • sparklyr from 1.7.7 to 1.7.8

    • stringr from 1.4.0 to 1.4.1

    • survival from 3.2-13 to 3.4-0

    • tinytex from 0.40 to 0.41

    • viridisLite from 0.4.0 to 0.4.1

  • Upgraded Java libraries:

    • com.fasterxml.jackson.core.jackson-annotations from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.core.jackson-core from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.core.jackson-databind from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.3 to 2.13.4

    • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.3 to 2.13.4

    • org.apache.hadoop.hadoop-client-api from 3.3.2-databricks to 3.3.4-databricks

    • org.apache.hadoop.hadoop-client-runtime from 3.3.2 to 3.3.4

    • org.apache.orc.orc-core from 1.7.5 to 1.7.6

    • org.apache.orc.orc-mapreduce from 1.7.5 to 1.7.6

    • org.apache.orc.orc-shims from 1.7.5 to 1.7.6

    • org.apache.parquet.parquet-column from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.apache.parquet.parquet-common from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.apache.parquet.parquet-encoding from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.apache.parquet.parquet-format-structures from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.apache.parquet.parquet-hadoop from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.apache.parquet.parquet-jackson from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

    • org.glassfish.jersey.containers.jersey-container-servlet from 2.34 to 2.36

    • org.glassfish.jersey.containers.jersey-container-servlet-core from 2.34 to 2.36

    • org.glassfish.jersey.core.jersey-client from 2.34 to 2.36

    • org.glassfish.jersey.core.jersey-common from 2.34 to 2.36

    • org.glassfish.jersey.core.jersey-server from 2.34 to 2.36

    • org.glassfish.jersey.inject.jersey-hk2 from 2.34 to 2.36

Apache Spark

Databricks Runtime 11.3 LTS includes Apache Spark 3.3.0. This release includes all Spark fixes and improvements included in Databricks Runtime 11.2, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-39957] [WARMFIX][SC-111425][CORE] Delay onDisconnected to enable Driver receives ExecutorExitCode

  • [SPARK-39955] [WARMFIX][SC-111424][CORE] Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

  • [SPARK-40474] [SC-106248][Cherry-Pick] Correct CSV schema inference behavior for datetime columns and introduce auto detection for Date fields

  • [SPARK-40535] [SC-111243][SQL] Fix bug the buffer of AggregatingAccumulator will not be created if the input rows is empty

  • [SPARK-40434] [SC-111125][SC-111144][SC-111138][SPARK-40435][11.3][SS][PYTHON] Implement applyInPandasWithState in PySpark

  • [SPARK-40460] [SC-110832][SS] Fix streaming metrics when selecting _metadata

  • [SPARK-40324] [SC-109943][SQL] Provide a query context of ParseException

  • [SPARK-40466] [SC-110899][SS] Improve the error message when DSv2 is disabled while DSv1 is not avaliable

  • [SPARK-40456] [SC-110848][SQL] PartitionIterator.hasNext should be cheap to call repeatedly

  • [SPARK-40169] [SC-110772][SQL] Don’t pushdown Parquet filters with no reference to data schema

  • [SPARK-40467] [SC-110759][SS] Split FlatMapGroupsWithState down to multiple test suites

  • [SPARK-40468] [SC-110813][SQL] Fix column pruning in CSV when corruptrecord is selected

  • [SPARK-40291] [SC-110085][SQL] Improve the message for column not in group by clause error

  • [SPARK-40398] [SC-110762][CORE][SQL] Use Loop instead of Arrays.stream api

  • [SPARK-40433] [SC-110684][SS][PYTHON] Add toJVMRow in PythonSQLUtils to convert pickled PySpark Row to JVM Row

  • [SPARK-40414] [SC-110568][SQL][PYTHON] More generic type on PythonArrowInput and PythonArrowOutput

  • [SPARK-40352] [SC-109945][SQL] Add function aliases: len, datepart, dateadd, date_diff and curdate

  • [SPARK-40470] [SC-110761][SQL] Handle GetArrayStructFields and GetMapValue in “arrays_zip” function

  • [SPARK-40387] [SC-110685][SQL] Improve the implementation of Spark Decimal

  • [SPARK-40429] [SC-110675][SQL] Only set KeyGroupedPartitioning when the referenced column is in the output

  • [SPARK-40432] [SC-110716][SS][PYTHON] Introduce GroupStateImpl and GroupStateTimeout in PySpark

  • [SPARK-39915] [SC-110496][SQL] Ensure the output partitioning is user-specified in AQE

  • [SPARK-29260] [SQL] Support ALTER DATABASE SET LOCATION if HMS supports

  • [SPARK-40185] [SC-110056][SQL] Remove column suggestion when the candidate list is empty

  • [SPARK-40362] [SC-110401][SQL] Fix BinaryComparison canonicalization

  • [SPARK-40411] [SC-110381][SS] Refactor FlatMapGroupsWithStateExec to have a parent trait

  • [SPARK-40293] [SC-110084][SQL] Make the V2 table error message more meaningful

  • [SPARK-38734] [SC-110383][SQL] Remove the error class INDEX_OUT_OF_BOUNDS

  • [SPARK-40292] [SC-110300][SQL] Fix column names in “arrays_zip” function when arrays are referenced from nested structs

  • [SPARK-40276] [SC-109674][CORE] Reduce the result size of RDD.takeOrdered

  • [SPARK-40197] [SC-109176][SQL] Replace query plan with context for MULTI_VALUE_SUBQUERY_ERROR

  • [SPARK-40300] [SC-109942][SQL] Migrate onto the DATATYPE_MISMATCH error class

  • [SPARK-40149] [SC-110055][SQL] Propagate metadata columns through Project

  • [SPARK-40280] [SC-110146][SQL] Add support for parquet push down for annotated int and long

  • [SPARK-40220] [SC-110143][SC-109175][SQL] Don’t output the empty map of error message parameters

  • [SPARK-40295] [SC-110070][SQL] Allow v2 functions with literal args in write distribution/ordering

  • [SPARK-40156] [SC-109264][SQL] url_decode() should the return an error class

  • [SPARK-39195] [SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

  • [SPARK-40260] [SC-109424][SQL] Use error classes in the compilation errors of GROUP BY a position

  • [SPARK-40205] [SC-110144][SC-109082][SQL] Provide a query context of ELEMENT_AT_BY_INDEX_ZERO

  • [SPARK-40112] [SC-109676][SQL] Improve the TO_BINARY() function

  • [SPARK-40209] [SC-109081][SQL] Don’t change the interval value of Decimal in changePrecision() on errors

  • [SPARK-40319] [SC-109873][SQL] Remove duplicated query execution error method for PARSE_DATETIME_BY_NEW_PARSER

  • [SPARK-40222] [SC-109209][SQL] Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children

  • [SPARK-40183] [SC-108907][SQL] Use error class NUMERIC_VALUE_OUT_OF_RANGE for overflow in decimal conversion

  • [SPARK-40180] [SC-109069][SQL] Format error messages by spark-sql

  • [SPARK-40153] [SC-109165][SQL] Unify resolve functions and table-valued functions

  • [SPARK-40308] [SC-109880][SQL] Allow non-foldable delimiter arguments to str_to_map function

  • [SPARK-40219] [SC-110052][SC-109663][SQL] Resolved view logical plan should hold the schema to avoid redundant lookup

  • [SPARK-40098] [SC-109939][SC-108693][SQL] Format error messages in the Thrift Server

  • [SPARK-39917] [SC-109038][SQL] Use different error classes for numeric/interval arithmetic overflow

  • [SPARK-40033] [SC-109875][SQL] Nested schema pruning support through element_at

  • [SPARK-40194] [SC-109660][SQL] SPLIT function on empty regex should truncate trailing empty string.

  • [SPARK-40228] [SC-109835][SQL] Do not simplify multiLike if child is not a cheap expression

  • [SPARK-40039] [SC-109896][SC-109260][SS] Introducing a streaming checkpoint file manager based on Hadoop’s Abortable interface

  • [SPARK-40285] [SC-109679][SQL] Simplify the roundTo[Numeric] for Spark Decimal

  • [SPARK-39896] [SC-109658][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed

  • [SPARK-40040] [SC-109662][SQL] Push local limit to both sides if join condition is empty

  • [SPARK-40055] [SC-109075][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog

  • [SPARK-39915] [SC-109391][SQL] Dataset.repartition(N) may not create N partitions Non-AQE part

  • [SPARK-40207] [SC-109401][SQL] Specify the column name when the data type is not supported by datasource

  • [SPARK-40245] [SC-109295][SQL] Fix FileScan equality check when partition or data filter columns are not read

  • [SPARK-40113] [SC-109405][SQL] Reactor ParquetScanBuilder DataSourceV2 interface implementations

  • [SPARK-40211] [SC-109226][CORE][SQL] Allow customize initial partitions number in take() behavior

  • [SPARK-40252] [SC-109379][SQL] Replace Stream.collect(Collectors.joining) with StringJoiner Api

  • [SPARK-40247] [SC-109272][SQL] Fix BitSet equality check

  • [SPARK-40067] [SQL] Use Table#name() instead of Scan#name() to populate the table name in the BatchScan node in SparkUI

  • [SPARK-39966] [SQL] Use V2 Filter in SupportsDelete

  • [SPARK-39607] [SC-109268][SQL][DSV2] Distribution and ordering support V2 function in writing

  • [SPARK-40224] [SC-109271][SQL] Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based

  • [SPARK-40013] [SQL] DS V2 expressions should have the default toString

  • [SPARK-40214] [SC-109079][PYTHON][SQL] add ‘get’ to functions

  • [SPARK-40192] [SC-109089][SQL][ML] Remove redundant groupby

  • [SPARK-40146] [SC-108694][SQL] Simply the codegen of getting map value

  • [SPARK-40109] [SQL] New SQL function: get()

  • [SPARK-39929] [SQL] DS V2 supports push down string functions(non ANSI)

  • [SPARK-39819] [SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions)

  • [SPARK-40213] [SC-109077][SQL] Support ASCII value conversion for Latin-1 characters

  • [SPARK-39887] [SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique

  • [SPARK-39764] [SQL] Make PhysicalOperation the same as ScanOperation

  • [SPARK-39964] [SQL] DS V2 pushdown should unify the translate path

  • [SPARK-39528] [SQL] Use V2 Filter in SupportsRuntimeFiltering

  • [SPARK-40066] [SQL] ANSI mode: always return null on invalid access to map column

  • [SPARK-39912] [SPARK-39828][SQL] Refine CatalogImpl

  • [SPARK-39833] [SC-108736][SQL] Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns

  • [SPARK-39880] [SQL] V2 SHOW FUNCTIONS command should print qualified function name like v1

  • [SPARK-39767] [SQL] Remove UnresolvedDBObjectName and add UnresolvedIdentifier

  • [SPARK-40163] [SC-108740][SQL] feat: SparkSession.config(Map)

  • [SPARK-40136] [SQL] Fix the fragment of SQL query contexts

  • [SPARK-40107] [SC-108689][SQL] Pull out empty2null conversion from FileFormatWriter

  • [SPARK-40121] [PYTHON][SQL] Initialize projection used for Python UDF

  • [SPARK-40128] [SQL] Make the VectorizedColumnReader recognize DELTA_LENGTH_BYTE_ARRAY as a standalone column encoding

  • [SPARK-40132] [ML] Restore rawPredictionCol to MultilayerPerceptronClassifier.setParams

  • [SPARK-40050] [SC-108696][SQL] Enhance EliminateSorts to support removing sorts via LocalLimit

  • [SPARK-39629] [SQL] Support v2 SHOW FUNCTIONS

  • [SPARK-39925] [SC-108734][SQL] Add array_sort(column, comparator) overload to DataFrame operations

  • [SPARK-40117] [PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite

  • [SPARK-40105] [SQL] Improve repartition in ReplaceCTERefWithRepartition

  • [SPARK-39503] [SQL] Add session catalog name for v1 database table and function

  • [SPARK-39889] [SQL] Use different error classes for numeric/interval divided by 0

  • [SPARK-39741] [SQL] Support url encode/decode as built-in function and tidy up url-related functions

  • [SPARK-40102] [SQL] Use SparkException instead of IllegalStateException in SparkPlan

  • [SPARK-40014] [SQL] Support cast of decimals to ANSI intervals

  • [SPARK-39776] [SQL][FOLLOW] Update UT of PlanStabilitySuite in ANSI mode

  • [SPARK-39963] [SQL] Simplify SimplifyCasts.isWiderCast

System environment

  • Operating System: Ubuntu 20.04.5 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.14

  • Python: 3.9.5

  • R: 4.1.3

  • Delta Lake: 2.1.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

argon2-cffi

20.1.0

async-generator

1.10

attrs

21.2.0

backcall

0.2.0

backports.entry-points-selectable

1.1.1

black

22.3.0

bleach

4.0.0

boto3

1.21.18

botocore

1.24.18

certifi

2021.10.8

cffi

1.14.6

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.3

cryptography

3.4.8

cycler

0.10.0

Cython

0.29.24

dbus-python

1.2.16

debugpy

1.4.1

decorator

5.1.0

defusedxml

0.7.1

distlib

0.3.6

entrypoints

0.3

facets-overview

1.0.0

filelock

3.8.0

idna

3.2

ipykernel

6.12.1

ipython

7.32.0

ipython-genutils

0.2.0

ipywidgets

7.7.0

jedi

0.18.0

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.0.1

jsonschema

3.2.0

jupyter-client

6.1.12

jupyter-core

4.8.1

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.1

MarkupSafe

2.0.1

matplotlib

3.4.3

matplotlib-inline

0.1.2

mistune

0.8.4

mypy-extensions

0.4.3

nbclient

0.5.3

nbconvert

6.1.0

nbformat

5.1.3

nest-asyncio

1.5.1

notebook

6.4.5

numpy

1.20.3

packaging

21.0

pandas

1.3.4

pandocfilters

1.4.3

parso

0.8.2

pathspec

0.9.0

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.4.0

pip

21.2.4

platformdirs

2.5.2

plotly

5.9.0

prometheus-client

0.11.0

prompt-toolkit

3.0.20

protobuf

4.21.5

psutil

5.8.0

psycopg2

2.9.3

ptyprocess

0.7.0

pyarrow

7.0.0

pycparser

2.20

Pygments

2.10.0

PyGObject

3.36.0

pyodbc

4.0.31

pyparsing

3.0.4

pyrsistent

0.18.0

python-dateutil

2.8.2

pytz

2021.3

pyzmq

22.2.1

requests

2.26.0

requests-unixsocket

0.2.0

s3transfer

0.5.2

scikit-learn

0.24.2

scipy

1.7.1

seaborn

0.11.2

Send2Trash

1.8.0

setuptools

58.0.4

six

1.16.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

terminado

0.9.4

testpath

0.5.0

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tomli

2.0.1

tornado

6.1

traitlets

5.1.0

typing-extensions

3.10.0.2

unattended-upgrades

0.1

urllib3

1.26.7

virtualenv

20.8.0

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.37.0

widgetsnbextension

3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-09-08.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.1.3

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-7

brio

1.1.3

broom

1.0.1

bslib

0.4.0

cachem

1.0.6

callr

3.7.2

caret

6.0-93

cellranger

1.1.0

chron

2.3-57

class

7.3-20

cli

3.3.0

clipr

0.8.0

cluster

2.1.3

codetools

0.2-18

colorspace

2.0-3

commonmark

1.8.0

compiler

4.1.3

config

0.3.1

cpp11

0.4.2

crayon

1.5.1

credentials

1.3.2

curl

4.3.2

data.table

1.14.2

datasets

4.1.3

DBI

1.1.3

dbplyr

2.2.1

desc

1.4.1

devtools

2.4.4

diffobj

0.3.5

digest

0.6.29

downlit

0.4.2

dplyr

1.0.10

dtplyr

1.2.2

e1071

1.7-11

ellipsis

0.3.2

evaluate

0.16

fansi

1.0.3

farver

2.1.1

fastmap

1.1.0

fontawesome

0.3.0

forcats

0.5.2

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.28.0

future.apply

1.9.1

gargle

1.2.0

generics

0.1.3

gert

1.8.0

ggplot2

3.3.6

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-4

globals

0.16.1

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.1

gower

1.0.0

graphics

4.1.3

grDevices

4.1.3

grid

4.1.3

gridExtra

2.3

gsubfn

0.7

gtable

0.3.1

hardhat

1.2.0

haven

2.5.1

highr

0.9

hms

1.1.2

htmltools

0.5.3

htmlwidgets

1.5.4

httpuv

1.6.5

httr

1.4.4

ids

1.0.1

ini

0.3.1

ipred

0.9-13

isoband

0.2.5

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.0

KernSmooth

2.23-20

knitr

1.40

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.6.10

lifecycle

1.0.1

listenv

0.8.0

lubridate

1.8.0

magrittr

2.0.3

markdown

1.1

MASS

7.3-56

Matrix

1.4-1

memoise

2.0.1

methods

4.1.3

mgcv

1.8-40

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.9

munsell

0.5.0

nlme

3.1-157

nnet

7.3-17

numDeriv

2016.8-1.1

openssl

2.0.2

parallel

4.1.3

parallelly

1.32.1

pillar

1.8.1

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgdown

2.0.6

pkgload

1.3.0

plogr

0.2.0

plyr

1.8.7

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.7.0

prodlim

2019.11.13

profvis

0.3.7

progress

1.2.2

progressr

0.11.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.1

purrr

0.3.4

r2d3

0.2.6

R6

2.5.1

ragg

1.2.2

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.9

RcppEigen

0.3.3.9.2

readr

2.1.2

readxl

1.4.1

recipes

1.0.1

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.0.5

rmarkdown

2.16

RODBC

1.3-19

roxygen2

7.2.1

rpart

4.1.16

rprojroot

2.0.3

Rserve

1.8-11

RSQLite

2.2.16

rstudioapi

0.14

rversions

2.1.2

rvest

1.0.3

sass

0.4.2

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.2

sourcetools

0.1.7

sparklyr

1.7.8

SparkR

3.3.0

spatial

7.3-11

splines

4.1.3

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.3

stats4

4.1.3

stringi

1.7.8

stringr

1.4.1

survival

3.4-0

sys

3.4

systemfonts

1.0.4

tcltk

4.1.3

testthat

3.1.4

textshaping

0.3.6

tibble

3.1.8

tidyr

1.2.0

tidyselect

1.1.2

tidyverse

1.3.2

timeDate

4021.104

tinytex

0.41

tools

4.1.3

tzdb

0.3.0

urlchecker

1.0.1

usethis

2.1.6

utf8

1.2.2

utils

4.1.3

uuid

1.1-0

vctrs

0.4.1

viridisLite

0.4.1

vroom

1.5.7

waldo

0.4.0

whisker

0.4

withr

2.5.0

xfun

0.32

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.5

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.189

com.amazonaws

aws-java-sdk-cloudformation

1.12.189

com.amazonaws

aws-java-sdk-cloudfront

1.12.189

com.amazonaws

aws-java-sdk-cloudhsm

1.12.189

com.amazonaws

aws-java-sdk-cloudsearch

1.12.189

com.amazonaws

aws-java-sdk-cloudtrail

1.12.189

com.amazonaws

aws-java-sdk-cloudwatch

1.12.189

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.189

com.amazonaws

aws-java-sdk-codedeploy

1.12.189

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.189

com.amazonaws

aws-java-sdk-cognitosync

1.12.189

com.amazonaws

aws-java-sdk-config

1.12.189

com.amazonaws

aws-java-sdk-core

1.12.189

com.amazonaws

aws-java-sdk-datapipeline

1.12.189

com.amazonaws

aws-java-sdk-directconnect

1.12.189

com.amazonaws

aws-java-sdk-directory

1.12.189

com.amazonaws

aws-java-sdk-dynamodb

1.12.189

com.amazonaws

aws-java-sdk-ec2

1.12.189

com.amazonaws

aws-java-sdk-ecs

1.12.189

com.amazonaws

aws-java-sdk-efs

1.12.189

com.amazonaws

aws-java-sdk-elasticache

1.12.189

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.189

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.189

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.189

com.amazonaws

aws-java-sdk-emr

1.12.189

com.amazonaws

aws-java-sdk-glacier

1.12.189

com.amazonaws

aws-java-sdk-glue

1.12.189

com.amazonaws

aws-java-sdk-iam

1.12.189

com.amazonaws

aws-java-sdk-importexport

1.12.189

com.amazonaws

aws-java-sdk-kinesis

1.12.189

com.amazonaws

aws-java-sdk-kms

1.12.189

com.amazonaws

aws-java-sdk-lambda

1.12.189

com.amazonaws

aws-java-sdk-logs

1.12.189

com.amazonaws

aws-java-sdk-machinelearning

1.12.189

com.amazonaws

aws-java-sdk-opsworks

1.12.189

com.amazonaws

aws-java-sdk-rds

1.12.189

com.amazonaws

aws-java-sdk-redshift

1.12.189

com.amazonaws

aws-java-sdk-route53

1.12.189

com.amazonaws

aws-java-sdk-s3

1.12.189

com.amazonaws

aws-java-sdk-ses

1.12.189

com.amazonaws

aws-java-sdk-simpledb

1.12.189

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.189

com.amazonaws

aws-java-sdk-sns

1.12.189

com.amazonaws

aws-java-sdk-sqs

1.12.189

com.amazonaws

aws-java-sdk-ssm

1.12.189

com.amazonaws

aws-java-sdk-storagegateway

1.12.189

com.amazonaws

aws-java-sdk-sts

1.12.189

com.amazonaws

aws-java-sdk-support

1.12.189

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.189

com.amazonaws

jmespath-java

1.12.189

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.13.4

com.fasterxml.jackson.core

jackson-core

2.13.4

com.fasterxml.jackson.core

jackson-databind

2.13.4

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.13.4

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.13.4

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.13.4

com.fasterxml.jackson.module

jackson-module-paranamer

2.13.4

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.13.4

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.2-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.1

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.0.204

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.ning

compress-lzf

1.1

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.0

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.5.1

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.74.Final

io.netty

netty-buffer

4.1.74.Final

io.netty

netty-codec

4.1.74.Final

io.netty

netty-common

4.1.74.Final

io.netty

netty-handler

4.1.74.Final

io.netty

netty-resolver

4.1.74.Final

io.netty

netty-tcnative-classes

2.0.48.Final

io.netty

netty-transport

4.1.74.Final

io.netty

netty-transport-classes-epoll

4.1.74.Final

io.netty

netty-transport-classes-kqueue

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-aarch_64

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-x86_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-aarch_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-x86_64

4.1.74.Final

io.netty

netty-transport-native-unix-common

4.1.74.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.13

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.2

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.14

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

7.0.0

org.apache.arrow

arrow-memory-core

7.0.0

org.apache.arrow

arrow-memory-netty

7.0.0

org.apache.arrow

arrow-vector

7.0.0

org.apache.avro

avro

1.11.0

org.apache.avro

avro-ipc

1.11.0

org.apache.avro

avro-mapred

1.11.0

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.9

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.4-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.4

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.14

org.apache.ivy

ivy

2.5.0

org.apache.logging.log4j

log4j-1.2-api

2.18.0

org.apache.logging.log4j

log4j-api

2.18.0

org.apache.logging.log4j

log4j-core

2.18.0

org.apache.logging.log4j

log4j-slf4j-impl

2.18.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.7.6

org.apache.orc

orc-mapreduce

1.7.6

org.apache.orc

orc-shims

1.7.6

org.apache.parquet

parquet-column

1.12.0-databricks-0007

org.apache.parquet

parquet-common

1.12.0-databricks-0007

org.apache.parquet

parquet-encoding

1.12.0-databricks-0007

org.apache.parquet

parquet-format-structures

1.12.0-databricks-0007

org.apache.parquet

parquet-hadoop

1.12.0-databricks-0007

org.apache.parquet

parquet-jackson

1.12.0-databricks-0007

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.46.v20220331

org.eclipse.jetty

jetty-continuation

9.4.46.v20220331

org.eclipse.jetty

jetty-http

9.4.46.v20220331

org.eclipse.jetty

jetty-io

9.4.46.v20220331

org.eclipse.jetty

jetty-jndi

9.4.46.v20220331

org.eclipse.jetty

jetty-plus

9.4.46.v20220331

org.eclipse.jetty

jetty-proxy

9.4.46.v20220331

org.eclipse.jetty

jetty-security

9.4.46.v20220331

org.eclipse.jetty

jetty-server

9.4.46.v20220331

org.eclipse.jetty

jetty-servlet

9.4.46.v20220331

org.eclipse.jetty

jetty-servlets

9.4.46.v20220331

org.eclipse.jetty

jetty-util

9.4.46.v20220331

org.eclipse.jetty

jetty-util-ajax

9.4.46.v20220331

org.eclipse.jetty

jetty-webapp

9.4.46.v20220331

org.eclipse.jetty

jetty-xml

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-api

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-client

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-common

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-server

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-servlet

9.4.46.v20220331

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.36

org.glassfish.jersey.containers

jersey-container-servlet-core

2.36

org.glassfish.jersey.core

jersey-client

2.36

org.glassfish.jersey.core

jersey-common

2.36

org.glassfish.jersey.core

jersey-server

2.36

org.glassfish.jersey.inject

jersey-hk2

2.36

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.mlflow

mlflow-spark

1.27.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.3

org.roaringbitmap

RoaringBitmap

0.9.25

org.roaringbitmap

shims

0.9.25

org.rocksdb

rocksdbjni

6.24.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.36

org.slf4j

jul-to-slf4j

1.7.36

org.slf4j

slf4j-api

1.7.36

org.spark-project.spark

unused

1.0.0

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1