Databricks Runtime 7.2 (Unsupported)

Databricks released this image in August 2020.

The following release notes provide information about Databricks Runtime 7.2, powered by Apache Spark 3.0.

New features

Databricks Runtime 7.2 includes the following new features.

Auto Loader is generally available

Auto Loader is an efficient method for incrementally ingesting a large number of files into Delta Lake. It is now GA and adds the following features: directory listing, cloud resource management API, rate limiting option, and option validation.

Directory listing mode option

Auto Loader adds a new directory listing mode, in addition to the existing file notification mode, for determining when there are new files.

  • Directory listing mode: In some scenarios there are only a few files that need to be streamed in on a regular basis. In those cases, identifying new files by parallel listing of the input directory and ingesting them is reasonably efficient. This allows you to start Auto Loader streams without any permission configuration. Directory listing mode is the default mode for Auto Loader from Databricks Runtime 7.2 and above.

  • File notification mode: You can explicitly opt in to file notification mode, which is more performant and scalable for large input directories, but requires you to configure permissions for the notification service to subscribe to file events and set .option("cloudFiles.useNotifications", "true"). Existing Auto Loader streams started on Databricks Runtime 7.1 and below automatically opt in to file notification mode when they upgrade to Databricks Runtime 7.2.

For more information, see What is Auto Loader?.

Cloud resource management API

You can now use our Scala API to manage cloud resources created by Auto Loader. You can list notification services and tear down specific notification services using this API. For more information, see Manually configure or manage file notification resources.

Rate limiting option

You can now use the cloudFiles.maxBytesPerTrigger option to limit the amount of data processed in each microbatch. You can specify a byte string such as 10g to limit each microbatch to 10 GB of data. This is a soft maximum. If you have files that are 3 GB each, Databricks processes 12 GB in a microbatch. This option has no effect when used with Trigger.Once(). When this option is not provided, the default value of cloudFiles.maxFilesPerTrigger of 1000 files is used for rate limiting. When used together with cloudFiles.maxFilesPerTrigger, Databricks consumes up to the lowest limit of cloudFiles.maxFilesPerTrigger or cloudFiles.maxBytesPerTrigger, whichever is reached first.

For more information, see What is Auto Loader?.

Option validation

Auto Loader now validates the options you provide. If you provide an unknown option, Auto Loader returns an error instead of silently ignoring the option. The validation also detects inconsistencies in your provided options. For example, if you set cloudFiles.useNotifications to false and at the same time provide permission options for AWS SNS and SQS services, validation will fail. To skip option validation, set cloudFiles.validateOptions to false.

Changes in default option values and compatibility

The default behavior of some Auto Loader options changed in Databricks Runtime 7.2. To ensure compatibility, the behavior of streams started on Databricks Runtime 7.1 and below will remain unchanged even after you upgrade to running such streams on a cluster running Databricks Runtime 7.2 or above. See What is Auto Loader?.

Efficiently copy a Delta table with clone (Public Preview)

You can now efficiently create a shallow or deep clone of a Delta table at a specific version with the clone command. A deep clone is a standalone clone of a Delta table formed by copying the files in parallel over from the source table to the target table. A shallow clone is a cheaply created clone that don’t copy files to the target table but points to files in the original table. Any changes made to a deep or shallow clone affects only the clone and not the original table. See Clone a table on Databricks.

Upgraded S3 connector offers improved stability and performance (Public Preview)

The upgraded connector is based on Hadoop 3.1.3 and contains many stability improvements (see Hadoop stability epics 2, 3 and 4), including optimizations on LIST that can lead to significant LIST cost reduction, depending on depth and breath of the S3 directory structure. For details, see Working with data in Amazon S3.

Improvements

  • Snowflake connector: The connector has been upgraded to version 2.8.1, which includes Spark 3.0 support.

  • PySpark usability: when a PySpark or pandas UDF fails, Databricks now surfaces the Python-side exception and most recently executed line of notebook code (if applicable) at the top of the error message.

  • Credential passthrough:

    • You can now use the LDA transform function on a passthrough-enabled cluster.

    • You can now set the storage checkpoint directory on a passthrough-enabled cluster.

  • Cluster libraries: Databricks now processes all cluster libraries in the order that they were installed on the cluster. On Databricks Runtime 7.1 and below, only Maven and CRAN libraries were processed in the order they were installed on the cluster. Other libraries were installed in a nondeterministic order.

  • TensorBoard:

    • Manage TensorBoard processes with the tensorboard.notebook API.

    • You can have multiple TensorBoard backends running on a single cluster.

    • In Python notebooks, display the TensorBoard UI inline using the %tensorboard line magic command.

    • dbutils.tensorboard is deprecated.

    See TensorBoard.

  • Upgraded Python libraries:

    • jmespath upgraded from 0.9.4 to 0.10.0.

    • koalas upgraded from 1.0.1 to 1.1.0.

  • Upgraded several installed R libraries. See Installed R Libraries.

Apache Spark

Databricks Runtime 7.2 includes Apache Spark 3.0.0. This release includes all Spark fixes and improvements included in Databricks Runtime 7.1 (Unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-32234] [SQL] Spark sql commands are failing on selecting the orc tables

  • [SPARK-30616] [SQL] Introduce TTL config option for SQL Metadata Cache

  • [SPARK-21117] [SQL] Built-in SQL Function Support - WIDTH_BUCKET

  • [SPARK-32168] [SQL] Fix hidden partitioning correctness bug in SQL overwrite

  • [SPARK-31935] Hadoop file system config should be effective in data source options

  • [SPARK-32270] [SQL] Use TextFileFormat in CSV’s schema inference with a different encoding

  • [SPARK-28481] [SQL] More expressions should extend NullIntolerant

  • [SPARK-32220] [SQL] SHUFFLE_REPLICATE_NL Hint should not change Non-Cartesian Product join result

  • [SPARK-32232] [ML][PYSPARK] Make sure ML has the same default solver values between Scala and Python

  • [SPARK-32238] [SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in ScalaUDF

  • [SPARK-32220] [SQL] SHUFFLE_REPLICATE_NL Hint should not change Non-Cartesian Product join result

  • [SPARK-32256] [SQL] Force to initialize Hadoop VersionInfo in HiveExternalCatalog

  • [SPARK-32159] [SQL] Fix integration between Aggregator[Array[_], _, _] and UnresolvedMapObjects

  • [SPARK-32160] [CORE][PYSPARK] Disallow to create SparkContext in executors

  • [SPARK-32168] [SQL] Fix hidden partitioning correctness bug in SQL overwrite

  • [SPARK-32024] [WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing

  • [SPARK-32214] [SQL] The type conversion function generated in makeFromJava for “other” type uses a wrong variable

  • [SPARK-30703] [SQL] Update SqlBase.g4 invalid comment

  • [SPARK-32163] [SQL] Nested pruning should work even with cosmetic variations

  • [SPARK-32124] [CORE] Use the invalid value Int.MinValue to fill the map index when the event logs from the old Spark version

  • [SPARK-32167] [SQL] Fix GetArrayStructFields to respect inner field’s nullability together

  • [SPARK-32130] [SQL] Enable timestamps inference in JsonBenchmark

  • [SPARK-32121] [SHUFFLE] Support Windows OS in ExecutorDiskUtils

  • [SPARK-32130] [SQL] Disable the JSON option inferTimestamp by default

  • [SPARK-31935] [SQL] Hadoop file system config should be effective in data source options

  • [SPARK-32095] [SQL] Update documentation to reflect usage of updated statistics

  • [SPARK-32131] [SQL] Fix AnalysisException messages at UNION/EXCEPT/MINUS operations

  • [SPARK-32068] [WEBUI] Correct task lauchtime show issue due to timezone in stage tab

  • [SPARK-29999] [SS] Fix test to check the actual metadata log directory

  • [SPARK-32127] [SQL] Check rules for MERGE INTO should use MergeAction.conditition other than MergeAction.children

  • [SPARK-32038] [SQL] Make the alias name pretty after float/double normalization

  • [SPARK-32126] [SS] Scope Session.active in IncrementalExecution

  • [SPARK-32115] [SQL] Fix SUBSTRING to handle integer overflows

  • [SPARK-32098] [PYTHON] Use iloc for positional slicing instead of direct slicing in createDataFrame with Arrow

  • [SPARK-32080] [SPARK-31998] [SQL] Simplify ArrowColumnVector ListArray accessor

  • [SPARK-32028] [WEBUI] fix app id link for multi attempts app in history summary page

  • [SPARK-31918] [R] Ignore S4 generic methods under SparkR namespace in closure cleaning to support R 4.0.0+

  • [SPARK-32038] [SQL] NormalizeFloatingNumbers should also work on distinct aggregate

  • [SPARK-32034] [SQL] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown

  • [SPARK-32021] [SQL] Increase precision of seconds and fractions of make_interval

  • [SPARK-31980] [SQL] Function sequence() fails if start and end of range are equal dates

  • [SPARK-32020] [SQL] Better error message when SPARK_HOME or spark.test.home is not set

  • [SPARK-32011] [PYTHON][CORE] Remove warnings about pin-thread modes and guide to use collectWithJobGroup

System environment

  • Operating System: Ubuntu 18.04.4 LTS

  • Java: 1.8.0_252

  • Scala: 2.12.10

  • Python: 3.7.5

  • R: R version 3.6.3 (2020-02-29)

  • Delta Lake 0.7.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

asn1crypto

1.3.0

backcall

0.1.0

boto3

1.12.0

botocore

1.15.0

certifi

2020.6.20

cffi

1.14.0

chardet

3.0.4

cryptography

2.8

cycler

0.10.0

Cython

0.29.15

decorator

4.4.1

docutils

0.15.2

entrypoints

0.3

idna

2.8

ipykernel

5.1.4

ipython

7.12.0

ipython-genutils

0.2.0

jedi

0.14.1

jmespath

0.10.0

joblib

0.14.1

jupyter-client

5.3.4

jupyter-core

4.6.1

kiwisolver

1.1.0

koalas

1.1.0

matplotlib

3.1.3

numpy

1.18.1

pandas

1.0.1

parso

0.5.2

patsy

0.5.1

pexpect

4.8.0

pickleshare

0.7.5

pip

20.0.2

prompt-toolkit

3.0.3

psycopg2

2.8.4

ptyprocess

0.6.0

pyarrow

0.15.1

pycparser

2.19

Pygments

2.5.2

PyGObject

3.26.1

pyOpenSSL

19.1.0

pyparsing

2.4.6

PySocks

1.7.1

python-apt

1.6.5+ubuntu0.3

python-dateutil

2.8.1

pytz

2019.3

pyzmq

18.1.1

requests

2.22.0

s3transfer

0.3.3

scikit-learn

0.22.1

scipy

1.4.1

seaborn

0.10.0

setuptools

45.2.0

six

1.14.0

ssh-import-id

5.7

statsmodels

0.11.0

tornado

6.0.3

traitlets

4.3.3

unattended-upgrades

0.1

urllib3

1.25.8

virtualenv

16.7.10

wcwidth

0.1.8

wheel

0.34.2

Installed R libraries

R libraries are installed from (Microsoft CRAN snapshot on 2020-07-14).

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.1.8

base

3.6.3

base64enc

0.1-3

BH

1.72.0-3

bit

1.1-15.2

bit64

0.9-7

blob

1.2.1

boot

1.3-25

brew

1.0-6

broom

0.7.0

callr

3.4.3

caret

6.0-86

cellranger

1.1.0

chron

2.3-55

class

7.3-17

cli

2.0.2

clipr

0.7.0

cluster

2.1.0

codetools

0.2-16

colorspace

1.4-1

commonmark

1.7

compiler

3.6.3

config

0.3

covr

3.5.0

crayon

1.3.4

crosstalk

1.1.0.1

curl

4.3

data.table

1.12.8

datasets

3.6.3

DBI

1.1.0

dbplyr

1.4.4

desc

1.2.0

devtools

2.3.0

digest

0.6.25

dplyr

0.8.5

DT

0.14

ellipsis

0.3.1

evaluate

0.14

fansi

0.4.1

farver

2.0.3

fastmap

1.0.1

forcats

0.5.0

foreach

1.5.0

foreign

0.8-76

forge

0.2.0

fs

1.4.2

generics

0.0.2

ggplot2

3.3.2

gh

1.1.0

git2r

0.27.1

glmnet

3.0-2

globals

0.12.5

glue

1.4.1

gower

0.2.2

graphics

3.6.3

grDevices

3.6.3

grid

3.6.3

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

haven

2.3.1

highr

0.8

hms

0.5.3

htmltools

0.5.0

htmlwidgets

1.5.1

httpuv

1.5.4

httr

1.4.1

hwriter

1.3.2

hwriterPlus

1.0-3

ini

0.3.1

ipred

0.9-9

isoband

0.2.2

iterators

1.0.12

jsonlite

1.7.0

KernSmooth

2.23-17

knitr

1.29

labeling

0.3

later

1.1.0.1

lattice

0.20-41

lava

1.6.7

lazyeval

0.2.2

lifecycle

0.2.0

lubridate

1.7.9

magrittr

1.5

markdown

1.1

MASS

7.3-51.6

Matrix

1.2-18

memoise

1.1.0

methods

3.6.3

mgcv

1.8-31

mime

0.9

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-147

nnet

7.3-14

numDeriv

2016.8-1.1

openssl

1.4.2

parallel

3.6.3

pillar

1.4.6

pkgbuild

1.1.0

pkgconfig

2.0.3

pkgload

1.1.0

plogr

0.2.0

plyr

1.8.6

praise

1.0.0

prettyunits

1.1.1

pROC

1.16.2

processx

3.4.3

prodlim

2019.11.13

progress

1.2.2

promises

1.1.1

proto

1.0.0

ps

1.3.3

purrr

0.3.4

r2d3

0.2.3

R6

2.4.1

randomForest

4.6-14

rappdirs

0.3.1

rcmdcheck

1.3.3

RColorBrewer

1.1-2

Rcpp

1.0.5

readr

1.3.1

readxl

1.3.1

recipes

0.1.13

rematch

1.0.1

rematch2

2.1.2

remotes

2.1.1

reprex

0.3.0

reshape2

1.4.4

rex

1.2.0

rjson

0.2.20

rlang

0.4.7

rmarkdown

2.3

RODBC

1.3-16

roxygen2

7.1.1

rpart

4.1-15

rprojroot

1.3-2

Rserve

1.8-7

RSQLite

2.2.0

rstudioapi

0.11

rversions

2.0.2

rvest

0.3.5

scales

1.1.1

selectr

0.4-2

sessioninfo

1.1.1

shape

1.4.4

shiny

1.5.0

sourcetools

0.1.7

sparklyr

1.3.1

SparkR

3.0.0

spatial

7.3-11

splines

3.6.3

sqldf

0.4-11

SQUAREM

2020.3

stats

3.6.3

stats4

3.6.3

stringi

1.4.6

stringr

1.4.0

survival

3.1-12

sys

3.3

tcltk

3.6.3

TeachingDemos

2.10

testthat

2.3.2

tibble

3.0.3

tidyr

1.1.0

tidyselect

1.1.0

tidyverse

1.3.0

timeDate

3043.102

tinytex

0.24

tools

3.6.3

usethis

1.6.1

utf8

1.1.4

utils

3.6.3

uuid

0.1-4

vctrs

0.3.1

viridisLite

0.3.0

whisker

0.4

withr

2.2.0

xfun

0.15

xml2

1.3.2

xopen

1.0.0

xtable

1.8-4

yaml

2.2.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.11.655

com.amazonaws

aws-java-sdk-cloudformation

1.11.655

com.amazonaws

aws-java-sdk-cloudfront

1.11.655

com.amazonaws

aws-java-sdk-cloudhsm

1.11.655

com.amazonaws

aws-java-sdk-cloudsearch

1.11.655

com.amazonaws

aws-java-sdk-cloudtrail

1.11.655

com.amazonaws

aws-java-sdk-cloudwatch

1.11.655

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.655

com.amazonaws

aws-java-sdk-codedeploy

1.11.655

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.655

com.amazonaws

aws-java-sdk-cognitosync

1.11.655

com.amazonaws

aws-java-sdk-config

1.11.655

com.amazonaws

aws-java-sdk-core

1.11.655

com.amazonaws

aws-java-sdk-datapipeline

1.11.655

com.amazonaws

aws-java-sdk-directconnect

1.11.655

com.amazonaws

aws-java-sdk-directory

1.11.655

com.amazonaws

aws-java-sdk-dynamodb

1.11.655

com.amazonaws

aws-java-sdk-ec2

1.11.655

com.amazonaws

aws-java-sdk-ecs

1.11.655

com.amazonaws

aws-java-sdk-efs

1.11.655

com.amazonaws

aws-java-sdk-elasticache

1.11.655

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.655

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.655

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.655

com.amazonaws

aws-java-sdk-emr

1.11.655

com.amazonaws

aws-java-sdk-glacier

1.11.655

com.amazonaws

aws-java-sdk-iam

1.11.655

com.amazonaws

aws-java-sdk-importexport

1.11.655

com.amazonaws

aws-java-sdk-kinesis

1.11.655

com.amazonaws

aws-java-sdk-kms

1.11.655

com.amazonaws

aws-java-sdk-lambda

1.11.655

com.amazonaws

aws-java-sdk-logs

1.11.655

com.amazonaws

aws-java-sdk-machinelearning

1.11.655

com.amazonaws

aws-java-sdk-opsworks

1.11.655

com.amazonaws

aws-java-sdk-rds

1.11.655

com.amazonaws

aws-java-sdk-redshift

1.11.655

com.amazonaws

aws-java-sdk-route53

1.11.655

com.amazonaws

aws-java-sdk-s3

1.11.655

com.amazonaws

aws-java-sdk-ses

1.11.655

com.amazonaws

aws-java-sdk-simpledb

1.11.655

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.655

com.amazonaws

aws-java-sdk-sns

1.11.655

com.amazonaws

aws-java-sdk-sqs

1.11.655

com.amazonaws

aws-java-sdk-ssm

1.11.655

com.amazonaws

aws-java-sdk-storagegateway

1.11.655

com.amazonaws

aws-java-sdk-sts

1.11.655

com.amazonaws

aws-java-sdk-support

1.11.655

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.655

com.amazonaws

jmespath-java

1.11.655

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.10.0

com.fasterxml.jackson.core

jackson-core

2.10.0

com.fasterxml.jackson.core

jackson-databind

2.10.0

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.10.0

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.10.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.10.0

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.10.0

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.joshelser

dropwizard-metrics-hadoop-metrics2-reporter

0.1.2

com.github.luben

zstd-jni

1.4.4-3

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.2.4

com.google.flatbuffers

flatbuffers-java

1.9.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

1.4.195

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.2.8

com.microsoft.sqlserver

mssql-jdbc

8.2.1.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.9.5

com.twitter

chill_2.12

0.9.5

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.8.3

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.9.4

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.10

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.47.Final

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.5

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

net.razorvine

pyrolite

4.30

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.12.8

net.snowflake

spark-snowflake_2.12

2.8.1-spark_3.0

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.7.1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

0.15.1

org.apache.arrow

arrow-memory

0.15.1

org.apache.arrow

arrow-vector

0.15.1

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.commons

commons-compress

1.8.1

org.apache.commons

commons-crypto

1.0.0

org.apache.commons

commons-lang3

3.9

org.apache.commons

commons-math3

3.4.1

org.apache.commons

commons-text

1.6

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.4

org.apache.hadoop

hadoop-auth

2.7.4

org.apache.hadoop

hadoop-client

2.7.4

org.apache.hadoop

hadoop-common

2.7.4

org.apache.hadoop

hadoop-hdfs

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.4

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.4

org.apache.hadoop

hadoop-yarn-api

2.7.4

org.apache.hadoop

hadoop-yarn-client

2.7.4

org.apache.hadoop

hadoop-yarn-common

2.7.4

org.apache.hadoop

hadoop-yarn-server-common

2.7.4

org.apache.hive

hive-beeline

2.3.7

org.apache.hive

hive-cli

2.3.7

org.apache.hive

hive-common

2.3.7

org.apache.hive

hive-exec-core

2.3.7

org.apache.hive

hive-jdbc

2.3.7

org.apache.hive

hive-llap-client

2.3.7

org.apache.hive

hive-llap-common

2.3.7

org.apache.hive

hive-metastore

2.3.7

org.apache.hive

hive-serde

2.3.7

org.apache.hive

hive-shims

2.3.7

org.apache.hive

hive-storage-api

2.7.1

org.apache.hive

hive-vector-code-gen

2.3.7

org.apache.hive.shims

hive-shims-0.23

2.3.7

org.apache.hive.shims

hive-shims-common

2.3.7

org.apache.hive.shims

hive-shims-scheduler

2.3.7

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.6

org.apache.httpcomponents

httpcore

4.4.12

org.apache.ivy

ivy

2.4.0

org.apache.orc

orc-core

1.5.10

org.apache.orc

orc-mapreduce

1.5.10

org.apache.orc

orc-shims

1.5.10

org.apache.parquet

parquet-column

1.10.1.2-databricks4

org.apache.parquet

parquet-common

1.10.1.2-databricks4

org.apache.parquet

parquet-encoding

1.10.1.2-databricks4

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1.2-databricks4

org.apache.parquet

parquet-jackson

1.10.1.2-databricks4

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.velocity

velocity

1.5

org.apache.xbean

xbean-asm7-shaded

4.15

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.4.14

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.18.v20190429

org.eclipse.jetty

jetty-continuation

9.4.18.v20190429

org.eclipse.jetty

jetty-http

9.4.18.v20190429

org.eclipse.jetty

jetty-io

9.4.18.v20190429

org.eclipse.jetty

jetty-jndi

9.4.18.v20190429

org.eclipse.jetty

jetty-plus

9.4.18.v20190429

org.eclipse.jetty

jetty-proxy

9.4.18.v20190429

org.eclipse.jetty

jetty-security

9.4.18.v20190429

org.eclipse.jetty

jetty-server

9.4.18.v20190429

org.eclipse.jetty

jetty-servlet

9.4.18.v20190429

org.eclipse.jetty

jetty-servlets

9.4.18.v20190429

org.eclipse.jetty

jetty-util

9.4.18.v20190429

org.eclipse.jetty

jetty-webapp

9.4.18.v20190429

org.eclipse.jetty

jetty-xml

9.4.18.v20190429

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.30

org.glassfish.jersey.containers

jersey-container-servlet-core

2.30

org.glassfish.jersey.core

jersey-client

2.30

org.glassfish.jersey.core

jersey-common

2.30

org.glassfish.jersey.core

jersey-server

2.30

org.glassfish.jersey.inject

jersey-hk2

2.30

org.glassfish.jersey.media

jersey-media-jaxb

2.30

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.6.6

org.json4s

json4s-core_2.12

3.6.6

org.json4s

json4s-jackson_2.12

3.6.6

org.json4s

json4s-scalap_2.12

3.6.6

org.lz4

lz4-java

1.7.1

org.mariadb.jdbc

mariadb-java-client

2.1.2

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.7.45

org.roaringbitmap

shims

0.7.45

org.rocksdb

rocksdbjni

6.2.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.10

org.scala-lang

scala-library_2.12

2.12.10

org.scala-lang

scala-reflect_2.12

2.12.10

org.scala-lang.modules

scala-collection-compat_2.12

2.1.1

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.0

org.scalanlp

breeze_2.12

1.0

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.30

org.slf4j

jul-to-slf4j

1.7.30

org.slf4j

slf4j-api

1.7.30

org.slf4j

slf4j-log4j12

1.7.30

org.spark-project.spark

unused

1.0.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.5

org.typelevel

algebra_2.12

2.0.0-M2

org.typelevel

cats-kernel_2.12

2.0.0-M4

org.typelevel

machinist_2.12

0.6.8

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0-M1

org.typelevel

spire-platform_2.12

0.17.0-M1

org.typelevel

spire-util_2.12

0.17.0-M1

org.typelevel

spire_2.12

0.17.0-M1

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.7.5

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52