Databricks Runtime 8.4 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 8.4 and Databricks Runtime 8.4 Photon, powered by Apache Spark 3.1.2. Databricks released this version in July 2021. Photon is in Public Preview.

New features and improvements

Improve cluster security by using credential passthrough with AWS Glue Data catalog

Credential passthrough is now supported when using AWS Glue Data Catalog as the external Hive metastore. Callers with different permissions can write to the AWS Glue Data Catalog from the same cluster. For more information, see Step 6: Launch a cluster with the Glue Catalog instance profile.

Delta Lake features and improvements

Delta table change data feed (GA)

The Delta table change data feed is now generally available. It represents the row level changes between different versions of the table. When enabled, additional information is recorded regarding row level changes for every write operation on the table. See Use Delta Lake change data feed on Databricks.

Load shared Delta tables easily with Databricks Runtime

The Apache Spark Connector for Delta Sharing 0.1.0 is now in Databricks Runtime. You can load a shared table using spark.read.format("deltaSharing").load(uri) directly without attaching the Delta Sharing Spark connector to your cluster.

More tables benefit from dynamic file pruning

The dynamic file pruning feature has been tuned to trigger on tables with fewer files. See Dynamic file pruning.

Better performance with automatic target file size tuning

The target file size for Delta tables is now automatically tuned based on the table size. Previously, the target file size for OPTIMIZE and OPTIMIZE ZORDER BY was 1GB. With autotuning based on table size, Delta tables up to 2.56TB will use 256MB as the target size. Tables larger than 10TB will use 1GB as before. Tables between these sizes will use target sizes that grow proportionally with the table size. See Configure Delta Lake to control data file size.

More ways to specify tables in DeltaTable.forName

DeltaTable.forName now supports using delta.`<path>` to identify tables.

Robust streaming multi-table writes using foreachBatch

Idempotent Delta streaming writes within the foreachBatch() command are now supported. For details, see Idempotent multi-table writes.

Improved read query performance in certain workloads due to tuned checkpoints

Delta Lake now tunes how frequently it does enhanced checkpoints. Instead of checkpointing at a fixed interval, Delta now dynamically adjusts the checkpoint frequency based on certain event triggers. This improves read query performance in workloads where some data skipping optimization couldn’t be applied before. To use these optimizations, upgrade your jobs that write to Delta Lake to Databricks Runtime 8.4. See Enable enhanced checkpoints for Structured Streaming queries.

Create GroupState to test user defined Structured Streaming functions

Until now, only the Structured Streaming engine could create instances of GroupState. Hence, any unit tests of the user-defined function required running a streaming query in Apache Spark.

Now you can create instances of GroupState using TestGroupState.create(…). This allow you to test a user-defined function in simple unit tests that do not require running Spark. See Test the mapGroupsWithState update function. Specifically, it produces instances of type TestGroupState that extends the interface GroupState with additional methods for introspecting the internal state after the user-defined function has been applied.

Auto Loader features and improvements

Configure backfilling to capture missed files

Auto Loader now supports performing backfills asynchronously to capture any files that could have been missed with file notifications. File storage systems and notification systems cannot guarantee 100% delivery of all file events. Therefore Databricks recommends enabling periodic backfills to capture all of your data with Auto Loader. Use the cloudFiles.backfillInterval option to schedule regular backfills over your data. See Common Auto Loader options.

Cross-account file notification access

Auto Loader now supports loading data across AWS accounts by using assuming a role. After setting the temporary security credentials created by AssumeRole, you can have Auto Loader load cloud files cross-accounts. See Securely ingest data in a different AWS account.

Bound storage footprint for large volume streams

You can now configure Auto Loader to expire and remove entries in RocksDB to bound its storage footprint in the checkpoint location. Databricks does not recommend that you use unless you are ingesting data in the order of millions of files an hour. Setting this option incorrectly or attempting to tune it can lead to many data quality issues, such as unprocessed files being ignored or duplicate of some files instead of exactly once processing. For details, see Event retention.

Simplified configuration with pathless support

S3 buckets

You can now provide the SQS queue that receives events from multiple paths or S3 buckets. If you provide the SQS queue URL, the path option is not required for this use case. Auto Loader constructs S3 paths using the bucket and key from the S3 events. If you want to read the files through DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes to DBFS. This is not required unless you’re accessing data in different accounts with AssumeRole.

See File notification options.

Azure Data Lake Storage Gen2 containers

You can now provide the Azure queue that receives events from multiple containers. If you provide the Azure queue name, the path option is not required. By default, Auto Loader constructs Azure Data Lake Storage Gen2 paths using the container and key in the file events. If you want to use WASB paths or DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes.

See File notification options.

Connector upgrades

  • The Snowflake Spark connector has been updated to v2.9.0.

  • KMS encryption is now supported in the UNLOAD statement of the Redshift connector.

Bug fixes

  • Fixed an issue for clusters enabled with table access control where select * from folder can show folder content even if the user has no file access permission.

  • Non-admin database owners are now able to drop non-owned tables in databases. This fixes the issue where database owners could not drop a database if non-owned tables existed in the database.

Library upgrades

  • Upgraded Python libraries:

    • certifi upgraded from 2020.12.5 to 2021.5.30

    • distill upgraded from 0.3.1 to 0.3.2

    • koalas upgraded from 1.8.0 to 1.8.1

    • protobuf upgraded from 3.17.0 to 3.17.3

  • Upgraded R libraries:

    • base from 4.0.4 to 4.1.0

    • boot from 1.3-27 to 1.3-28

    • class from 7.3-18 to 7.3-19

    • cluster from 2.1.1 to 2.1.2

    • compiler from 4.0.4 to 4.1.0

    • datasets from 4.0.4 to 4.1.0

    • graphics from 4.0.4 to 4.1.0

    • grDevices from 4.0.4 to 4.1.0

    • grid from 4.0.4 to 4.1.0

    • KernSmooth from 2.23-18 to 2.23-20

    • lattice from 0.20-41 to 0.20-44

    • MASS from 7.3-53.1 to 7.3-54

    • Matrix from 1.3-2 to 1.3-3

    • methods from 4.0.4 to 4.1.0

    • mgcv from 1.8-33 to 1.8-35

    • nnet from 7.3-15 to 7.3-16

    • parallel from 4.0.4 to 4.1.0

    • Rserve from 1.8-7 to 1.8-8

    • SparkR from 3.1.1 to 3.1.2

    • splines from 4.0.4 to 4.1.0

    • stats from 4.0.4 to 4.1.0

    • stats4 from 4.0.4 to 4.1.0

    • survival from 3.2-7 to 3.2-11

    • tcltk from 4.0.4 to 4.1.0

    • tools from 4.0.4 to 4.1.0

    • utils from 4.0.4 to 4.1.0

  • Upgraded Java libraries:

    • snowflake-jdbc from 3.12.8 to 3.13.3

    • spark-snowflake_2.12 from 2.8.1-spark_3.0 to 2.9.0-spark_3.1

    • RoaringBitmap from 0.9.0 to 0.9.14

    • shims from 0.9.0 to 0.9.14

    • rocksdbjni fromm 6.2.2 to 6.20.3

Apache Spark

Databricks Runtime 8.4 includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 8.3 (EoS), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-35792] [SQL] View should not capture configs used in RelationConversions

  • [SPARK-35700] [SQL] Read char/varchar orc table with created and written by external systems

  • [SPARK-35636] [SQL] Lambda keys should not be referenced outside of the lambda function

  • [SPARK-35800] [Cherry Pick] Improving GroupState testability by introducing TestGroupState

  • [SPARK-35391] Fix memory leak in ExecutorAllocationListener

  • [SPARK-35799] [CherryPick] Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec

  • [SPARK-35763] [SS] Remove the StateStoreCustomMetric subclass enumeration dependency

  • [SPARK-35791 [SQL] Release on-going map properly for NULL-aware ANTI join

  • [SPARK-35695] [SQL] Collect observed metrics from cached and adaptive execution sub-trees

  • [SPARK-35767] [SQL] Avoid executing child plan twice in CoalesceExec

  • [SPARK-35746] [UI] Fix taskid in the stage page task event timeline

  • [SPARK-35673] [SQL] Fix user-defined hint and unrecognized hint in subquery.

  • [SPARK-35714] [CORE] Bug fix for deadlock during the executor shutdown

  • [SPARK-35689] [SS] Add log warn when keyWithIndexToValue returns null value

  • [SPARK-35589] [CORE][3.1] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

  • [SPARK-35643] [PYTHON] Fix ambiguous reference in functions.py column()

  • [SPARK-35652] [SQL] joinWith on two table generated from same one

  • [SPARK-35679] [SQL] instantToMicros overflow

  • [SPARK-35602] [SS] Update state schema to be able to accept long length JSON

  • [SPARK-35653] [SQL] Fix CatalystToExternalMap interpreted path fails for Map with case classes as keys or values

  • [SPARK-35296] [SQL] Allow Dataset.observe to work even if CollectMetricsExec in a task handles multiple partitions.

  • [SPARK-35659] [SS] Avoid write null to StateStore

  • [SPARK-35665] [SQL] Resolve UnresolvedAlias in CollectMetrics

  • [SPARK-35558] Optimizes for multi-quantile retrieval

  • [SPARK-35621] [SQL] Add rule id pruning to the TypeCoercion rule

  • [SPARK-35077] [SQL] Migrate to transformWithPruning for leftover optimizer rules

  • [SPARK-35610] [CORE] Fix the memory leak introduced by the Executor’s stop shutdown hook

  • [SPARK-35544] [SQL] Add tree pattern pruning to Analyzer rules

  • [SPARK-35566] [SS] Fix StateStoreRestoreExec output rows

  • [SPARK-35454] [SQL][3.1] One LogicalPlan can match multiple dataset ids

  • [SPARK-35538] [SQL] Migrate transformAllExpressions call sites to use transformAllExpressionsWithPruning

  • [SPARK-35106] [Core][SQL] Avoid failing rename caused by destination directory not exist

  • [SPARK-35287] [SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation

  • [SPARK-35495] [R] Change SparkR maintainer for CRAN

  • [SPARK-27991] [CORE] Defer the fetch request on Netty OOM

  • [SPARK-35171] [R] Declare the markdown package as a dependency of the SparkR package

  • [SPARK-35454] [SQL] One LogicalPlan can match multiple dataset ids

  • [SPARK-35298] [SQL] Migrate to transformWithPruning for rules in Optimizer.scala

  • [SPARK-35480] [SQL] Make percentile_approx work with pivot

  • [SPARK-35093] [SQL] AQE now uses newQueryStage plan as key for looking up cached exchanges for re-use

  • [SPARK-35146] [SQL] Migrate to transformWithPruning or resolveWithPruning for rules in finishAnalysis.scala

  • [SPARK-35411] [SQL] Add essential information while serializing TreeNode to json

  • [SPARK-35294] [SQL] Add tree traversal pruning in rules with dedicated files under optimizer

  • [SPARK-34897] [SQL][3.1] Support reconcile schemas based on index after nested column pruning

  • [SPARK-35144] [SQL] Migrate to transformWithPruning for object rules

  • [SPARK-35155] [SQL] Add rule id pruning to Analyzer rules

  • [SPARK-35382] [PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs.

  • [SPARK-35359] [SQL] Insert data with char/varchar datatype will fail when data length exceed length limitation

  • [SPARK-35381] [R] Fix lambda variable name issues in nested higher order functions at R APIs

System environment

  • Operating System: Ubuntu 18.04.5 LTS

  • Java: Zulu 8.54.0.21-CA-linux64

  • Scala: 2.12.10

  • Python: 3.8.8

  • R: 4.1.0 (2021-05-18)

  • Delta Lake 1.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

appdirs

1.4.4

asn1crypto

1.4.0

backcall

0.2.0

boto3

1.16.7

botocore

1.19.7

brotlipy

0.7.0

certifi

2021.5.30

cffi

1.14.3

chardet

3.0.4

cryptography

3.1.1

cycler

0.10.0

Cython

0.29.21

decorator

4.4.2

distlib

0.3.2

docutils

0.15.2

entrypoints

0.3

facets-overview

1.0.0

filelock

3.0.12

idna

2.10

ipykernel

5.3.4

ipython

7.19.0

ipython-genutils

0.2.0

jedi

0.17.2

jmespath

0.10.0

joblib

0.17.0

jupyter-client

6.1.7

jupyter-core

4.6.3

kiwisolver

1.3.0

koalas

1.8.1

matplotlib

3.2.2

numpy

1.19.2

pandas

1.1.5

parso

0.7.0

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

pip

20.2.4

plotly

4.14.3

prompt-toolkit

3.0.8

protobuf

3.17.3

psycopg2

2.8.5

ptyprocess

0.6.0

pyarrow

1.0.1

pycparser

2.20

Pygments

2.7.2

pyOpenSSL

19.1.0

pyparsing

2.4.7

PySocks

1.7.1

python-dateutil

2.8.1

pytz

2020.5

pyzmq

19.0.2

requests

2.24.0

retrying

1.3.3

s3transfer

0.3.6

scikit-learn

0.23.2

scipy

1.5.2

seaborn

0.10.0

setuptools

50.3.1

six

1.15.0

statsmodels

0.12.0

threadpoolctl

2.1.0

tornado

6.0.4

traitlets

5.0.5

urllib3

1.25.11

virtualenv

20.2.1

wcwidth

0.2.5

wheel

0.35.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.2.1

base

4.1.0

base64enc

0.1-3

BH

1.72.0-3

bit

4.0.4

bit64

4.0.5

blob

1.2.1

boot

1.3-28

brew

1.0-6

brio

1.1.0

broom

0.7.2

callr

3.5.1

caret

6.0-86

cellranger

1.1.0

chron

2.3-56

class

7.3-19

cli

2.2.0

clipr

0.7.1

cluster

2.1.2

codetools

0.2-18

colorspace

2.0-0

commonmark

1.7

compiler

4.1.0

config

0.3

covr

3.5.1

cpp11

0.2.4

crayon

1.3.4

credentials

1.3.0

crosstalk

1.1.0.1

curl

4.3

data.table

1.13.4

datasets

4.1.0

DBI

1.1.0

dbplyr

2.0.0

desc

1.2.0

devtools

2.3.2

diffobj

0.3.2

digest

0.6.27

dplyr

1.0.2

DT

0.16

ellipsis

0.3.1

evaluate

0.14

fansi

0.4.1

farver

2.0.3

fastmap

1.0.1

forcats

0.5.0

foreach

1.5.1

foreign

0.8-81

forge

0.2.0

fs

1.5.0

future

1.21.0

generics

0.1.0

gert

1.0.2

ggplot2

3.3.2

gh

1.2.0

gitcreds

0.1.1

glmnet

4.0-2

globals

0.14.0

glue

1.4.2

gower

0.2.2

graphics

4.1.0

grDevices

4.1.0

grid

4.1.0

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.3.1

highr

0.8

hms

0.5.3

htmltools

0.5.0

htmlwidgets

1.5.3

httpuv

1.5.4

httr

1.4.2

hwriter

1.3.2

hwriterPlus

1.0-3

ini

0.3.1

ipred

0.9-9

isoband

0.2.3

iterators

1.0.13

jsonlite

1.7.2

KernSmooth

2.23-20

knitr

1.30

labeling

0.4.2

later

1.1.0.1

lattice

0.20-44

lava

1.6.8.1

lazyeval

0.2.2

lifecycle

0.2.0

listenv

0.8.0

lubridate

1.7.9.2

magrittr

2.0.1

markdown

1.1

MASS

7.3-54

Matrix

1.3-3

memoise

1.1.0

methods

4.1.0

mgcv

1.8-35

mime

0.9

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-152

nnet

7.3-16

numDeriv

2016.8-1.1

openssl

1.4.3

parallel

4.1.0

parallelly

1.22.0

pillar

1.4.7

pkgbuild

1.1.0

pkgconfig

2.0.3

pkgload

1.1.0

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.16.2

processx

3.4.5

prodlim

2019.11.13

progress

1.2.2

promises

1.1.1

proto

1.0.0

ps

1.5.0

purrr

0.3.4

r2d3

0.2.3

R6

2.5.0

randomForest

4.6-14

rappdirs

0.3.1

rcmdcheck

1.3.3

RColorBrewer

1.1-2

Rcpp

1.0.5

readr

1.4.0

readxl

1.3.1

recipes

0.1.15

rematch

1.0.1

rematch2

2.1.2

remotes

2.2.0

reprex

0.3.0

reshape2

1.4.4

rex

1.2.0

rlang

0.4.9

rmarkdown

2.6

RODBC

1.3-17

roxygen2

7.1.1

rpart

4.1-15

rprojroot

2.0.2

Rserve

1.8-8

RSQLite

2.2.1

rstudioapi

0.13

rversions

2.0.2

rvest

0.3.6

scales

1.1.1

selectr

0.4-2

sessioninfo

1.1.1

shape

1.4.5

shiny

1.5.0

sourcetools

0.1.7

sparklyr

1.5.2

SparkR

3.1.1

spatial

7.3-11

splines

4.1.0

sqldf

0.4-11

SQUAREM

2020.5

stats

4.1.0

stats4

4.1.0

stringi

1.5.3

stringr

1.4.0

survival

3.2-11

sys

3.4

tcltk

4.1.0

TeachingDemos

2.10

testthat

3.0.0

tibble

3.0.4

tidyr

1.1.2

tidyselect

1.1.0

tidyverse

1.3.0

timeDate

3043.102

tinytex

0.28

tools

4.1.0

usethis

2.0.0

utf8

1.1.4

utils

4.1.0

uuid

0.1-4

vctrs

0.3.5

viridisLite

0.3.0

waldo

0.2.3

whisker

0.4

withr

2.3.0

xfun

0.19

xml2

1.3.2

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

zip

2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-glue

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-marketplacemeteringservice

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.10.0

com.fasterxml.jackson.core

jackson-core

2.10.0

com.fasterxml.jackson.core

jackson-databind

2.10.0

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.10.0

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.10.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.10.0

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.10.0

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.joshelser

dropwizard-metrics-hadoop-metrics2-reporter

0.1.2

com.github.luben

zstd-jni

1.4.8-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.2.4

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

9.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.9.5

com.twitter

chill_2.12

0.9.5

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.9.4

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

hive-2.3__hadoop-2.7

jets3t-0.7

liball_deps_2.12

hive-2.3__hadoop-2.7

zookeeper-3.4

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.10

io.delta

delta-sharing-spark_2.12

0.1.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.51.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.5

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

maven-trees

hive-2.3__hadoop-2.7

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.3

net.snowflake

spark-snowflake_2.12

2.9.0-spark_3.1

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8-1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

2.0.0

org.apache.arrow

arrow-memory-core

2.0.0

org.apache.arrow

arrow-memory-netty

2.0.0

org.apache.arrow

arrow-vector

2.0.0

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.commons

commons-compress

1.20

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.10

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.4

org.apache.hadoop

hadoop-auth

2.7.4

org.apache.hadoop

hadoop-client

2.7.4

org.apache.hadoop

hadoop-common

2.7.4

org.apache.hadoop

hadoop-hdfs

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.4

org.apache.hadoop

hadoop-yarn-api

2.7.4

org.apache.hadoop

hadoop-yarn-client

2.7.4

org.apache.hadoop

hadoop-yarn-common

2.7.4

org.apache.hadoop

hadoop-yarn-server-common

2.7.4

org.apache.hive

hive-beeline

2.3.7

org.apache.hive

hive-cli

2.3.7

org.apache.hive

hive-jdbc

2.3.7

org.apache.hive

hive-llap-client

2.3.7

org.apache.hive

hive-llap-common

2.3.7

org.apache.hive

hive-serde

2.3.7

org.apache.hive

hive-shims

2.3.7

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.7

org.apache.hive.shims

hive-shims-common

2.3.7

org.apache.hive.shims

hive-shims-scheduler

2.3.7

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.6

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.4.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.5.12

org.apache.orc

orc-mapreduce

1.5.12

org.apache.orc

orc-shims

1.5.12

org.apache.parquet

parquet-column

1.10.1-databricks9

org.apache.parquet

parquet-common

1.10.1-databricks9

org.apache.parquet

parquet-encoding

1.10.1-databricks9

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1-databricks9

org.apache.parquet

parquet-jackson

1.10.1-databricks9

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm7-shaded

4.15

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.4.14

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.36.v20210114

org.eclipse.jetty

jetty-continuation

9.4.36.v20210114

org.eclipse.jetty

jetty-http

9.4.36.v20210114

org.eclipse.jetty

jetty-io

9.4.36.v20210114

org.eclipse.jetty

jetty-jndi

9.4.36.v20210114

org.eclipse.jetty

jetty-plus

9.4.36.v20210114

org.eclipse.jetty

jetty-proxy

9.4.36.v20210114

org.eclipse.jetty

jetty-security

9.4.36.v20210114

org.eclipse.jetty

jetty-server

9.4.36.v20210114

org.eclipse.jetty

jetty-servlet

9.4.36.v20210114

org.eclipse.jetty

jetty-servlets

9.4.36.v20210114

org.eclipse.jetty

jetty-util

9.4.36.v20210114

org.eclipse.jetty

jetty-util-ajax

9.4.36.v20210114

org.eclipse.jetty

jetty-webapp

9.4.36.v20210114

org.eclipse.jetty

jetty-xml

9.4.36.v20210114

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.30

org.glassfish.jersey.containers

jersey-container-servlet-core

2.30

org.glassfish.jersey.core

jersey-client

2.30

org.glassfish.jersey.core

jersey-common

2.30

org.glassfish.jersey.core

jersey-server

2.30

org.glassfish.jersey.inject

jersey-hk2

2.30

org.glassfish.jersey.media

jersey-media-jaxb

2.30

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M5

org.json4s

json4s-core_2.12

3.7.0-M5

org.json4s

json4s-jackson_2.12

3.7.0-M5

org.json4s

json4s-scalap_2.12

3.7.0-M5

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.2.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.9.14

org.roaringbitmap

shims

0.9.14

org.rocksdb

rocksdbjni

6.20.3

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.10

org.scala-lang

scala-library_2.12

2.12.10

org.scala-lang

scala-reflect_2.12

2.12.10

org.scala-lang.modules

scala-collection-compat_2.12

2.1.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.0

org.scalanlp

breeze_2.12

1.0

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.5

org.typelevel

algebra_2.12

2.0.0-M2

org.typelevel

cats-kernel_2.12

2.0.0-M4

org.typelevel

machinist_2.12

0.6.8

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0-M1

org.typelevel

spire-platform_2.12

0.17.0-M1

org.typelevel

spire-util_2.12

0.17.0-M1

org.typelevel

spire_2.12

0.17.0-M1

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.2

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52