Databricks Runtime 10.3 (Unsupported)

The following release notes provide information about Databricks Runtime 10.3 and Databricks Runtime 10.3 Photon, powered by Apache Spark 3.2.1. Photon is in Public Preview. Databricks released these images in February 2022.

New features and improvements

Parquet and Avro writers now use the session time zone when rebasing timestamps in Parquet data sources

Parquet and Avro writers now use the session time zone while timestamp rebasing in legacy mode instead of using the default JVM time zone. The session time zone is set to the JVM time zone by default.

New functions in Spark SQL

The following Spark SQL functions are now supported in this release:

  • aes_decrypt: decrypts input by using AES.

  • aes_encrypt: encrypts input by using AES.

  • contains: determines whether an expression contains another expression.

  • endswith: determines whether an expression ends with another expression.

  • percentile_cont: returns a percentile value.

  • startswith: determines whether an expression starts with another expression.

Low shuffle merge is now generally available

The Delta MERGE INTO command uses a technique called low shuffle merge, which reduces shuffling of unmodified rows. This improves performance of the command and helps to preserve existing clustering on the table, such as Z-ordering. This functionality was introduced in Databricks Runtime 9.0 for Public Preview and has now reached general availability. To enable low shuffle merge, set spark.databricks.delta.merge.enableLowShuffle to true. See Low shuffle merge on Databricks.

New COPY INTO features: validation mode and schema evolution

Validation mode: COPY INTO now supports a VALIDATE mode where the data that is to be loaded into a table is validated but not written to the table. These validations include:

  • Whether the data can be parsed.

  • Whether the schema matches that of the table or if the schema needs to be evolved.

  • Whether all nullability and check constraints are met.

Example usage:

COPY INTO <table>
FROM <path_to_files>
FILEFORMAT = <format>
VALIDATE [ALL | <num_rows> ROWS]

The default is to validate all of the data that is to be loaded. You can provide a number of rows to be validated with the ROWS keyword, such as VALIDATE 15 ROWS. The COPY INTO statement returns a preview of the data of 50 rows or less, when a number of less than 50 is used with the ROWS keyword).

Schema evolution: COPY INTO now supports schema evolution with the mergeSchema option in COPY_OPTIONS. To evolve the schema of a table, you must have OWN permissions on the table.

Example usage:

COPY INTO <table>
FROM <path_to_files>
FILEFORMAT = <format>
COPY_OPTIONS ('mergeSchema' = 'true')

See COPY INTO.

Identity columns are now supported in Delta tables (Public Preview)

Delta Lake now supports identity columns. When you write to a Delta table that defines an identity column, and you do not provide values for that column, Delta now automatically assigns a unique and statistically increasing or decreasing value. See CREATE TABLE [USING].

Library upgrades

  • Upgraded Python libraries:

    • distlib from 0.3.3 to 0.3.4

    • filelock from 3.4.0 to 3.4.2

    • numpy from 1.19.2 to 1.20.1

    • plotly from 5.3.1 to 5.5.0

  • Upgraded R libraries:

    • backports from 1.4.0 to 1.4.1

    • brio from 1.1.2 to 1.1.3

    • broom from 0.7.10 to 0.7.11

    • class from 7.3-19 to 7.3-20

    • cpp11 from 0.4.1 to 0.4.2

    • DBI from 1.1.1 to 1.1.2

    • devtools from 2.4.2 to 2.4.3

    • digest from 0.6.28 to 0.6.29

    • dtplyr from 1.1.0 to 1.2.0

    • fansi from 0.5.0 to 1.0.2

    • foreign from 0.8-81 to 0.8-82

    • fs from 1.5.0 to 1.5.2

    • gert from 1.4.3 to 1.5.0

    • glue from 1.5.0 to 1.6.0

    • httpuv from 1.6.3 to 1.6.5

    • jsonlite from 1.7.2 to 1.7.3

    • knitr from 1.36 to 1.37

    • MASS from 7.3-54 to 7.3-55

    • Matrix from 1.3-4 to 1.4-0

    • nlme from 3.1-152 to 3.1-155

    • nnet frm 7.3-16 to 7.3-17

    • openssl from 1.4.5 to 1.4.6

    • parallelly from 1.29.0 to 1.30.0

    • pkgbuild from 1.2.0 to 1.3.1

    • pkgload from 1.2.3 to 1.2.4

    • progressr from 0.9.0 to 0.10.0

    • Rcpp from 1.0.7 to 1.0.8

    • readr from 2.1.0 to 2.1.1

    • remotes from 2.4.1 to 2.4.2

    • RSQLite from 2.2.8 to 2.2.9

    • sessioninfo from 1.2.1 to 1.2.2

    • sparklyr from 1.7.2 to 1.7.4

    • testthat from 3.1.0 to 3.1.1

    • tinytex from 0.35 to 0.36

    • usethis from 2.1.3 to 2.1.5

    • vroom from 1.5.6 to 1.5.7

    • withr from 2.4.2 to 2.4.3

    • xfun from 0.28 to 0.29

    • xml2 from 1.3.2 to 1.3.3

  • Upgraded Java libraries:

    • dev.ludovic.netlib.arpack from 1.3.2 to 2.2.1

    • dev.ludovic.netlib.blas from 1.3.2 to 2.2.1

    • dev.ludovic.netlib.lapack from 1.3.2 to 2.2.1

    • io.delta.delta-sharing-spark_2.12 from 0.2.0 to 0.3.0

Apache Spark

Databricks Runtime 10.3 includes Apache Spark 3.2.1. This release includes all Spark fixes and improvements included in Databricks Runtime 10.2 (Unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-37896] [SQL] Implement a ConstantColumnVector and improve performance of the hidden file metadata

  • [SPARK-37924] [SQL] Sort table properties by key in SHOW CREATE TABLE on VIEW (v1)

  • [SPARK-37636] [SQL] Migrate CREATE NAMESPACE to use V2 command by default

  • [SPARK-37381] [SQL] Unify v1 and v2 SHOW CREATE TABLE tests

  • [SPARK-37876] [CORE][SQL] Move SpecificParquetRecordReaderBase.listDirectory to TestUtils

  • [SPARK-37854] [CORE] Replace type check with pattern matching in Spark code

  • [SPARK-36967] [CORE] Report accurate shuffle block size if its skewed

  • [SPARK-37864] [SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2

  • [SPARK-37862] [SQL] RecordBinaryComparator should fast skip the check of aligning with unaligned platform

  • [SPARK-37855] [SQL] IllegalStateException when transforming an array inside a nested struct

  • [SPARK-37858] [SQL] Throw SparkRuntimeException with error classes from AES functions

  • [SPARK-37841] [SQL] BasicWriteTaskStatsTracker should not try get status for a skipped file

  • [SPARK-37860] [UI] Fix taskindex in the stage page task event timeline

  • [SPARK-36879] [SQL] Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path

  • [SPARK-37828] [SQL] Push down filters through RebalancePartitions

  • [SPARK-37802] [SQL][3.2] Composite field name should work with Aggregate push down

  • [SPARK-37826] [SQL] Use zstd codec name in ORC file names for hive orc impl

  • [SPARK-37820] [SQL] Replace ApacheCommonBase64 with JavaBase64 for string funcs

  • [SPARK-37832] [SQL] Orc struct converter should use an array to look up field converters rather than a linked list

  • [SPARK-37786] [SQL] StreamingQueryListener support use SQLConf.get to get corresponding SessionState’s SQLConf

  • [SPARK-37527] [SQL] Translate more standard aggregate functions for pushdown

  • [SPARK-37777] [SQL] Update the SQL syntax of SHOW FUNCTIONS

  • [SPARK-8582] [CORE] Checkpoint eagerly when asked to do so for real

  • [SPARK-37822] [SQL] StringSplit should return an array on non-null elements

  • [SPARK-37812] [SQL] When deserializing an Orc struct, reuse the result row when possible

  • [SPARK-37367] [SQL] Reenable exception test in DDLParserSuite.create view – basic

  • [SPARK-37796] [SQL] ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform

  • [SPARK-37817] [SQL][Minor] Remove unreachable code in complexTypeExtractors.scala

  • [SPARK-37807] [SQL] Fix a typo in HttpAuthenticationException message

  • [SPARK-37750] [SQL] ANSI mode: optionally return null result if element not exists in array/map

  • [SPARK-37789] [SQL] Add a class to represent general aggregate functions in DS V2

  • [SPARK-37785] [SQL][CORE] Add Utils.isInRunningSparkTask

  • [SPARK-37780] [SQL] QueryExecutionListener support use SQLConf.get to get corresponding SessionState’s SQLConf

  • [SPARK-37783] [SS][SQL][CORE] Enable tail-recursion wherever possible

  • [SPARK-37726] [SQL] Add spill size metrics for sort merge join

  • [SPARK-37800] [SQL] TreeNode.argString incorrectly formats arguments of type Set[_]

  • [SPARK-37716] [SQL] Improve error messages when a LateralJoin has non-deterministic expressions

  • [SPARK-37779] [SQL] Make ColumnarToRowExec plan canonicalizable after (de)serialization

  • [SPARK-37578] [SQL] Update task metrics from ds v2 custom metrics

  • [SPARK-37676] [SQL] Support ANSI Aggregation Function: percentile_cont

  • [SPARK-37724] [SQL] ANSI mode: disable ANSI reserved keywords by default

  • [SPARK-37705] [SQL][3.2] Rebase timestamps in the session time zone saved in Parquet/Avro metadata

  • [SPARK-34755] [SQL] Support the utils for transform number format

  • [SPARK-36703] [SQL] Remove the Sort if it is the child of RepartitionByExpression

  • [SPARK-37369] [SQL] Avoid redundant ColumnarToRow transistion on InMemoryTableScan

  • [SPARK-37450] [SQL] Prune unnecessary fields from Generate

  • [SPARK-37728] [SQL] Reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

  • [SPARK-37564] [SQL] Add code-gen for sort aggregate without grouping keys

  • [SPARK-37659] [UI][3.2] Fix FsHistoryProvider race condition between list and delet log info

  • [SPARK-37727] [SQL] Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate

  • [SPARK-37391] [SQL]JdbcConnectionProvider tells if it modifies security context

  • [SPARK-37273] [SQL] Support hidden file metadata columns in Spark SQL

  • [SPARK-36876] [SQL] Support Dynamic Partition pruning for HiveTableScanExec

  • [SPARK-37546] [SQL] V2 ReplaceTableAsSelect command should qualify location

  • [SPARK-37545] [SQL] V2 CreateTableAsSelect command should qualify location

  • [SPARK-37707] [SQL] Allow store assignment and implicit cast among datetime types

  • [SPARK-37644] [SQL] Support datasource v2 complete aggregate pushdown

  • [SPARK-37695] [CORE][SHUFFLE] Skip diagnosis ob merged blocks from push-based shuffle

  • [SPARK-35295] [ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0

  • [SPARK-37694] [SQL] Disallow delete resources in spark sql cli

  • [SPARK-37330] [SQL] Migrate ReplaceTableStatement to v2 command

  • [SPARK-27974] [SQL] Support ANSI Aggregate Function: array_agg

  • [SPARK-36902] [SQL] Migrate CreateTableAsSelectStatement to v2 command

  • [SPARK-37678] [PYTHON] Fix cleanupand_return signature

  • [SPARK-37613] [SQL] Support ANSI Aggregate Function: regr_count

  • [SPARK-37689] [SQL] Expand should be supported in PropagateEmptyRelation

  • [SPARK-37654] [SQL] Fix NPE in Row.getSeq when field is Null

  • [SPARK-37496] [SQL] Migrate ReplaceTableAsSelectStatement to v2 command

  • [SPARK-37483] [SQL] Support push down top N to JDBC data source V2

  • [SPARK-37577] [SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning

  • [SPARK-37310] [SQL] Migrate ALTER NAMESPACE … SET PROPERTIES to use V2 command by default

  • [SPARK-37501] [SQL] CREATE/REPLACE TABLE should qualify location for v2 command

  • [SPARK-37343] [SQL] Implement createIndex, IndexExists and dropIndex in JDBC (Postgres dialect)

  • [SPARK-37060] [CORE][3.2] Handle driver status response from backup masters

  • [SPARK-37629] [SQL] Speed up Expression.canonicalized

  • [SPARK-37633] [SQL] Unwrap cast should skip if downcast failed with an…

  • [SPARK-37631] [SQL] Code clean up on promoting strings to double type in math functions

  • [SPARK-37569] [SQL] Don’t mark nested view fields as nullable

  • [SPARK-37557] [SQL] Replace object hash with sort aggregate if child is already sorted

  • [SPARK-36850] [SQL] Migrate CreateTableStatement to v2 command framework

  • [SPARK-37592] [SQL] Improve performance of JoinSelection

  • [SPARK-37217] [SQL] The number of dynamic partitions should early check when writing to external tables

  • [SPARK-37584] [SQL] New SQL function: map_contains_key

  • [SPARK-37455] [SQL] Replace hash with sort aggregate if child is already sorted

  • [SPARK-34735] [SQL][UI] Add modified configs for SQL execution in UI

  • [SPARK-37540] [SQL] Detect more unsupported time travel

  • [SPARK-37376] [SQL] Introduce a new DataSource V2 interface HasPartitionKey

  • [SPARK-37666] [SQL] Set GCM as the default mode in aes_encrypt()/aes_decrypt()

  • [SPARK-37573] [SQL] IsolatedClientLoader should fallback to related version, not always hadoop-2.7.4

  • [SPARK-37561] [SQL] Avoid loading all functions when obtaining hive’s DelegationToken

  • [SPARK-37370] [SQL] Add SQL configs to control newly added join code-gen in 3.3

  • [SPARK-37311] [SQL] Migrate ALTER NAMESPACE … SET LOCATION to use V2 command by default

  • [SPARK-37192] [SQL] Migrate SHOW TBLPROPERTIES to use V2 command by default

  • [SPARK-37219] [SQL] Add AS OF syntax support

  • [SPARK-37316] [SQL] Add code-gen for existence sort merge join

  • [SPARK-37591] [SQL] Support the GCM mode by aes_encrypt()/aes_decrypt()

  • [SPARK-37586] [SQL] Add the mode and padding args to aes_encrypt()/aes_decrypt()

  • [SPARK-12567] [SQL] Add aes_encrypt and aes_decrypt builtin functions

  • [SPARK-37392] [SQL] Fix the performance bug when inferring constraints for Generate

  • [SPARK-37481] [Core][WebUI] Fix disappearance of skipped stages after they retry

  • [SPARK-37533] [SQL] New SQL function: try_element_at

  • [SPARK-37451] [SQL] Fix cast string type to decimal type if spark.sql.legacy.allowNegativeScaleOfDecimal is enabled

  • [SPARK-37062] [SS] Introduce a new data source for providing consistent set of rows per microbatch

  • [SPARK-37552] [SQL] Add the convert_timezone() function

  • [SPARK-37555] [SQL] spark-sql should pass last unclosed comment to backend

  • [SPARK-37286] [SQL] Move compileAggregates from JDBCRDD to JdbcDialect

  • [SPARK-37485] [CORE][SQL] Replace map with expressions which produce no result with foreach

  • [SPARK-37556] [SQL] Deser void class fail with Java serialization

  • [SPARK-37520] [SQL] Add the startswith() and endswith() string functions

  • [SPARK-37471] [SQL] spark-sql support ; in nested bracketed comment

  • [SPARK-37360] [SQL] Support TimestampNTZ in JSON data source

  • [SPARK-37508] [SQL] Add CONTAINS() string function

  • [SPARK-37494] [SQL] Unify v1 and v2 options output of SHOW CREATE TABLE command

  • [SPARK-37524] [SQL] We should drop all tables after testing dynamic partition pruning.

  • [SPARK-37389] [SQL] Check unclosed bracketed comments

  • [SPARK-37220] [SQL] Do not split input file for Parquet reader with aggregate push down

  • [SPARK-37326] [SQL] Support TimestampNTZ in CSV data source

  • [SPARK-35867] [SQL] Enable vectorized read for VectorizedPlainValuesReader.readBooleans

  • [SPARK-37484] [CORE][SQL] Replace get and getOrElse with getOrElse

  • [SPARK-37492] [SQL] Optimize Orc test code with withAllNativeOrcReaders

  • [SPARK-37458] [SS] Remove unnecessary SerializeFromObject from the plan of foreachBatch

System environment

  • Operating System: Ubuntu 20.04.3 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.14

  • Python: 3.8.10

  • R: 4.1.2

  • Delta Lake: 1.1.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

appdirs

1.4.4

argon2-cffi

20.1.0

async-generator

1.10

attrs

20.3.0

backcall

0.2.0

bidict

0.21.4

bleach

3.3.0

boto3

1.16.7

botocore

1.19.7

certifi

2020.12.5

cffi

1.14.5

chardet

4.0.0

cycler

0.10.0

Cython

0.29.23

dbus-python

1.2.16

decorator

5.0.6

defusedxml

0.7.1

distlib

0.3.4

distro-info

0.23ubuntu1

entrypoints

0.3

facets-overview

1.0.0

filelock

3.4.2

idna

2.10

ipykernel

5.3.4

ipython

7.22.0

ipython-genutils

0.2.0

ipywidgets

7.6.3

jedi

0.17.2

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.0.1

jsonschema

3.2.0

jupyter-client

6.1.12

jupyter-core

4.7.1

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.1

koalas

1.8.2

MarkupSafe

2.0.1

matplotlib

3.4.2

mistune

0.8.4

nbclient

0.5.3

nbconvert

6.0.7

nbformat

5.1.3

nest-asyncio

1.5.1

notebook

6.3.0

numpy

1.20.1

packaging

20.9

pandas

1.2.4

pandocfilters

1.4.3

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.2.0

pip

21.0.1

plotly

5.5.0

prometheus-client

0.10.1

prompt-toolkit

3.0.17

protobuf

3.17.2

psycopg2

2.8.5

ptyprocess

0.7.0

pyarrow

4.0.0

pycparser

2.20

Pygments

2.8.1

PyGObject

3.36.0

pyparsing

2.4.7

pyrsistent

0.17.3

python-apt

2.0.0+ubuntu0.20.4.6

python-dateutil

2.8.1

python-engineio

4.3.0

python-socketio

5.4.1

pytz

2020.5

pyzmq

20.0.0

requests

2.25.1

requests-unixsocket

0.2.0

s3transfer

0.3.7

scikit-learn

0.24.1

scipy

1.6.2

seaborn

0.11.1

Send2Trash

1.5.0

setuptools

52.0.0

six

1.15.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

terminado

0.9.4

testpath

0.4.4

threadpoolctl

2.1.0

tornado

6.1

traitlets

5.0.5

unattended-upgrades

0.1

urllib3

1.25.11

virtualenv

20.4.1

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.36.2

widgetsnbextension

3.5.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-01-19.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.1.2

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.2

boot

1.3-28

brew

1.0-6

brio

1.1.3

broom

0.7.11

bslib

0.3.1

cachem

1.0.6

callr

3.7.0

caret

6.0-90

cellranger

1.1.0

chron

2.3-56

class

7.3-20

cli

3.1.0

clipr

0.7.1

cluster

2.1.2

codetools

0.2-18

colorspace

2.0-2

commonmark

1.7

compiler

4.1.2

config

0.3.1

cpp11

0.4.2

crayon

1.4.2

credentials

1.3.2

curl

4.3.2

data.table

1.14.2

datasets

4.1.2

DBI

1.1.2

dbplyr

2.1.1

desc

1.4.0

devtools

2.4.3

diffobj

0.3.5

digest

0.6.29

dplyr

1.0.7

dtplyr

1.2.0

e1071

1.7-9

ellipsis

0.3.2

evaluate

0.14

fansi

1.0.2

farver

2.1.0

fastmap

1.1.0

fontawesome

0.2.2

forcats

0.5.1

foreach

1.5.1

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.23.0

future.apply

1.8.1

gargle

1.2.0

generics

0.1.1

gert

1.5.0

ggplot2

3.3.5

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-3

globals

0.14.0

glue

1.6.0

googledrive

2.0.0

googlesheets4

1.0.0

gower

0.2.2

graphics

4.1.2

grDevices

4.1.2

grid

4.1.2

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.4.3

highr

0.9

hms

1.1.1

htmltools

0.5.2

htmlwidgets

1.5.4

httpuv

1.6.5

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ids

1.0.1

ini

0.3.1

ipred

0.9-12

isoband

0.2.5

iterators

1.0.13

jquerylib

0.1.4

jsonlite

1.7.3

KernSmooth

2.23-20

knitr

1.37

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.6.10

lifecycle

1.0.1

listenv

0.8.0

lubridate

1.8.0

magrittr

2.0.1

markdown

1.1

MASS

7.3-55

Matrix

1.4-0

memoise

2.0.1

methods

4.1.2

mgcv

1.8-38

mime

0.12

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-155

nnet

7.3-17

numDeriv

2016.8-1.1

openssl

1.4.6

parallel

4.1.2

parallelly

1.30.0

pillar

1.6.4

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgload

1.2.4

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.5.2

prodlim

2019.11.13

progress

1.2.2

progressr

0.10.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-26

ps

1.6.0

purrr

0.3.4

r2d3

0.2.5

R6

2.5.1

randomForest

4.6-14

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-2

Rcpp

1.0.8

RcppEigen

0.3.3.9.1

readr

2.1.1

readxl

1.3.1

recipes

0.1.17

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.1

reshape2

1.4.4

rlang

0.4.12

rmarkdown

2.11

RODBC

1.3-19

roxygen2

7.1.2

rpart

4.1-15

rprojroot

2.0.2

Rserve

1.8-10

RSQLite

2.2.9

rstudioapi

0.13

rversions

2.1.1

rvest

1.0.2

sass

0.4.0

scales

1.1.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.1

sourcetools

0.1.7

sparklyr

1.7.4

SparkR

3.2.0

spatial

7.3-11

splines

4.1.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.2

stats4

4.1.2

stringi

1.7.6

stringr

1.4.0

survival

3.2-13

sys

3.4

tcltk

4.1.2

TeachingDemos

2.10

testthat

3.1.1

tibble

3.1.6

tidyr

1.1.4

tidyselect

1.1.1

tidyverse

1.3.1

timeDate

3043.102

tinytex

0.36

tools

4.1.2

tzdb

0.2.0

usethis

2.1.5

utf8

1.2.2

utils

4.1.2

uuid

1.0-3

vctrs

0.3.8

viridisLite

0.4.0

vroom

1.5.7

waldo

0.3.1

whisker

0.4

withr

2.4.3

xfun

0.29

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-glue

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.12.3

com.fasterxml.jackson.core

jackson-core

2.12.3

com.fasterxml.jackson.core

jackson-databind

2.12.3

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.12.3

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.12.3

com.fasterxml.jackson.module

jackson-module-paranamer

2.12.3

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.12.3

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.0-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.0

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

9.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.istack

istack-commons-runtime

3.0.8

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.8.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

hive-2.3__hadoop-3.2

jets3t-0.7

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.3.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.68.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.10

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

maven-trees

hive-2.3__hadoop-3.2

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.3

net.snowflake

spark-snowflake_2.12

2.9.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.10.2

org.apache.avro

avro-ipc

1.10.2

org.apache.avro

avro-mapred

1.10.2

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.1-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.1

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.htrace

htrace-core4

4.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.5.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.6.12

org.apache.orc

orc-mapreduce

1.6.12

org.apache.orc

orc-shims

1.6.12

org.apache.parquet

parquet-column

1.12.0-databricks-0003

org.apache.parquet

parquet-common

1.12.0-databricks-0003

org.apache.parquet

parquet-encoding

1.12.0-databricks-0003

org.apache.parquet

parquet-format-structures

1.12.0-databricks-0003

org.apache.parquet

parquet-hadoop

1.12.0-databricks-0003

org.apache.parquet

parquet-jackson

1.12.0-databricks-0003

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.43.v20210629

org.eclipse.jetty

jetty-continuation

9.4.43.v20210629

org.eclipse.jetty

jetty-http

9.4.43.v20210629

org.eclipse.jetty

jetty-io

9.4.43.v20210629

org.eclipse.jetty

jetty-jndi

9.4.43.v20210629

org.eclipse.jetty

jetty-plus

9.4.43.v20210629

org.eclipse.jetty

jetty-proxy

9.4.43.v20210629

org.eclipse.jetty

jetty-security

9.4.43.v20210629

org.eclipse.jetty

jetty-server

9.4.43.v20210629

org.eclipse.jetty

jetty-servlet

9.4.43.v20210629

org.eclipse.jetty

jetty-servlets

9.4.43.v20210629

org.eclipse.jetty

jetty-util

9.4.43.v20210629

org.eclipse.jetty

jetty-util-ajax

9.4.43.v20210629

org.eclipse.jetty

jetty-webapp

9.4.43.v20210629

org.eclipse.jetty

jetty-xml

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-api

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-client

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-common

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-server

9.4.43.v20210629

org.eclipse.jetty.websocket

websocket-servlet

9.4.43.v20210629

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jaxb

jaxb-runtime

2.3.2

org.glassfish.jersey.containers

jersey-container-servlet

2.34

org.glassfish.jersey.containers

jersey-container-servlet-core

2.34

org.glassfish.jersey.core

jersey-client

2.34

org.glassfish.jersey.core

jersey-common

2.34

org.glassfish.jersey.core

jersey-server

2.34

org.glassfish.jersey.inject

jersey-hk2

2.34

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.2.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.2.19

org.roaringbitmap

RoaringBitmap

0.9.14

org.roaringbitmap

shims

0.9.14

org.rocksdb

rocksdbjni

6.20.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1