Databricks Runtime 5.1 (Unsupported)

Databricks released this image in December 2018.

The following release notes provide information about Databricks Runtime 5.1, powered by Apache Spark.

New features

  • Delta Lake

    • Support for extended syntax in MERGE INTO.

      • Support for multiple MATCHED clauses, additional conditions in MATCHED and NOT MATCHED clauses, DELETE action. This is useful for writing changed data (inserts, updates, and deletes) from a database to a Delta table.

      • Support for * in UPDATE and INSERT actions to automatically fill in column names (similar to * in SELECT). This makes it easier to write MERGE queries for tables with a very large number of columns.

      For examples, see MERGE INTO.

    • Support for tunable isolation levels for Delta tables.

      • Delta tables can be configured to have one of these two isolation levels: Serializable and WriteSerializable (default).

      • The default isolation level is relaxed from Serializable to WriteSerializable to allow for greater availability. Specifically, UPDATE, DELETE, and MERGE are allowed to run concurrently with appends (for example, from a streaming query). See Isolation levels to understand the implications of this relaxation and the steps to enforce previous behavior if needed.

    • Private Preview of CONVERT TO DELTA command, which allows you to convert a Parquet table to a Delta table in-place, without copying any files. Contact Databricks support if you are interested in trying it out.

  • Notebook-scoped libraries for Python. You can now install Python libraries, including Eggs, Wheels, and PyPI, into Python notebooks using the Databricks Utilities library API. Libraries installed through this API are available only in the notebook in which the libraries are installed. See Library utility (dbutils.library).

    Preview

    This feature is in Public Preview.

    Library utilities enabled by default on clusters running Databricks Runtime 5.1. Therefore, by default the Python REPL process for each notebook is isolated by using a separate Python executable created when the notebook is attached and inherits the default Python environment on the cluster. Libraries installed through init scripts into a Databricks Python environment are still available. This feature can be disabled by setting spark.databricks.libraryIsolation.enabled to false.

  • The display function now supports pandas DataFrames. See display function.

Improvements

  • Delta Lake

    • Multi-cluster writes now support JDBC/ODBC queries.

    • Improved scalability of OPTIMIZE with a large number of small files.

    • You can now SET a nested column in UPDATE.

  • SQL Performance

    • Optimize LIMIT by fetching only the first n partitions.

    • Optimize range joins. We have added optimizer rules that rewrite a range join to be executed in a more efficient way. This can speed up this type of query by an order of magnitude. To enable the optimization, you add a hint to the query. The current rewrite supports the following range join query shapes:

      • Point in interval: a.begin < b.point AND b.point < a.end

      • Interval overlap: a.begin < b.end AND b.begin < a.end

  • Upgraded the connector for Azure Data Lake Storage Gen2, which includes critical bug fixes. We recommend that all users of Azure Data Lake Storage Gen2 upgrade to Databricks Runtime 5.1.

  • Upgraded Java from 1.8.0_162 to 1.8.0_181.

  • Upgraded Snowflake Connector for Spark to 2.4.10.

  • Upgraded some installed Python libraries:

    • pip: 18.0 to 18.1

    • setuptools: 40.4.1 to 40.6.2

    • virtualenv: 15.0.1 to 16.1.0

    • wheel: 0.31.1 to 0.32.3

  • Upgraded several installed R libraries. See Installed R Libraries.

Bug fixes

  • Fixed bug in Azure Data Lake Storage Gen2 support that failed to list directories with more than 5000 files.

  • TRANSFORM should handle different input data types correctly.

  • Clean up expired external shuffle files after executors die to prevent long-running clusters from running out of disk.

Apache Spark

Databricks Runtime 5.1 includes Apache Spark 2.4.0. This release includes all fixes and improvements included in Databricks Runtime 5.0 (Unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-26147][SQL] only pull out unevaluable python udf from join condition

  • [SPARK-26129][SQL] Instrumentation for per-query planning time

  • [SPARK-25921][PYSPARK] Fix barrier task run without BarrierTaskContext while python worker reuse

  • [SPARK-23207][SQL] Use SQLConf.get.enableRadixSort instead of SparkEnv.get.conf.get(SQLConf.RADIXSORT_ENABLED)_

  • [SPARK-26057][SQL] Transform also analyzed plans when dedup references

  • [SPARK-25974][SQL] Optimizes Generates bytecode for ordering based on the given order

  • [SPARK-25901][CORE] Use only one thread in BarrierTaskContext compani…

  • [SPARK-25381][SQL] Stratified sampling by Column argument

  • [SPARK-25514][SQL] Generating pretty JSON by to_json

  • [SPARK-22827][SQL] Throw SparkOutOfMemoryError in HashAggregateExec, too.

  • [SPARK-25979][SQL] Window function: allow parentheses around window reference

  • [SPARK-25988][SQL] Keep names unchanged when deduplicating the column names in Analyzer

  • [SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue

  • [SPARK-24901][SQL] Merge the codegen of RegularHashMap and fastHashMap to reduce compiler maxCodesize when VectorizedHashMap is false.

  • [SPARK-25850][SQL] Make the split threshold for the code generated function configurable

  • [SPARK-25866][ML] Update KMeans formatVersion

  • [SPARK-22148][SPARK-15815][SCHEDULER] Acquire new executors to avoid hang because of blocklisting

  • [SPARK-25827][CORE] Avoid converting incoming encrypted blocks to byte buffers

  • [SPARK-25918][SQL] LOAD DATA LOCAL INPATH should handle a relative path

  • [SPARK-25837][CORE] Fix potential slowdown in AppStatusListener when cleaning up stages

  • [SPARK-25773][CORE] Cancel zombie tasks in a result stage when the job finishes (5.x)

  • [SPARK-25767][SQL] Fix lazily evaluated stream of expressions in code generation

  • [SPARK-25819][SQL] Support parse mode option for the function fromavro_

  • [SPARK-25044] Change ScalaUDF constructor signature

  • [SPARK-25699][SQL] Partially push down conjunctive predicated in ORC

  • [SPARK-25559][SQL] Remove the unsupported predicates in Parquet when …

  • [SPARK-25540][SPARK-25525][SQL][PYSPARK] Do not update conf for existing SparkContext in SparkSession.getOrCreate.

  • [SPARK-25595] Ignore corrupt Avro files if flag IGNORE_CORRUPT_FILES enabled

  • [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TABLE

  • [SPARK-25449][CORE] Heartbeat shouldn’t include accumulators for zero metrics

  • [SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs

  • [SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file

System environment

  • Operating System: Ubuntu 16.04.5 LTS

  • Java: 1.8.0_181

  • Scala: 2.11.8

  • Python: 2.7.12 for Python 2 clusters and 3.5.2 for Python 3 clusters.

  • R: R version 3.4.4 (2018-03-15)

  • GPU clusters: The following NVIDIA GPU libraries are installed:

    • Tesla driver 375.66

    • CUDA 9.0

    • cuDNN 7.0

Note

Although Scala 2.12 is supported in Apache Spark 2.4, it is not supported in Databricks Runtime 5.1.

Installed Python libraries

Library

Version

Library

Version

Library

Version

ansi2html

1.1.1

argparse

1.2.1

backports-abc

0.5

boto

2.42.0

boto3

1.4.1

botocore

1.4.70

brewer2mpl

1.4.1

certifi

2016.2.28

cffi

1.7.0

chardet

2.3.0

colorama

0.3.7

configobj

5.0.6

cryptography

1.5

cycler

0.10.0

Cython

0.24.1

decorator

4.0.10

docutils

0.14

enum34

1.1.6

et-xmlfile

1.0.1

freetype-py

1.0.2

funcsigs

1.0.2

fusepy

2.0.4

futures

3.2.0

ggplot

0.6.8

html5lib

0.999

idna

2.1

ipaddress

1.0.16

ipython

2.2.0

ipython-genutils

0.1.0

jdcal

1.2

Jinja2

2.8

jmespath

0.9.0

llvmlite

0.13.0

lxml

3.6.4

MarkupSafe

0.23

matplotlib

1.5.3

mpld3

0.2

msgpack-python

0.4.7

ndg-httpsclient

0.3.3

numba

0.28.1

numpy

1.11.1

openpyxl

2.3.2

pandas

0.19.2

pathlib2

2.1.0

patsy

0.4.1

pexpect

4.0.1

pickleshare

0.7.4

Pillow

3.3.1

pip

18.1

ply

3.9

prompt-toolkit

1.0.7

psycopg2

2.6.2

ptyprocess

0.5.1

py4j

0.10.3

pyarrow

0.8.0

pyasn1

0.1.9

pycparser

2.14

Pygments

2.1.3

PyGObject

3.20.0

pyOpenSSL

16.0.0

pyparsing

2.2.0

pypng

0.0.18

Python

2.7.12

python-dateutil

2.5.3

python-geohash

0.8.5

pytz

2016.6.1

requests

2.11.1

s3transfer

0.1.9

scikit-learn

0.18.1

scipy

0.18.1

scour

0.32

seaborn

0.7.1

setuptools

40.6.2

simplejson

3.8.2

simples3

1.0

singledispatch

3.4.0.3

six

1.10.0

statsmodels

0.6.1

tornado

5.1.1

traitlets

4.3.0

urllib3

1.19.1

virtualenv

16.1.0

wcwidth

0.1.7

wheel

0.32.3

wsgiref

0.1.2

Installed R libraries

Library

Version

Library

Version

Library

Version

abind

1.4-5

assertthat

0.2.0

backports

1.1.2

base

3.4.4

base64enc

0.1-3

BH

1.66.0-1

bindr

0.1.1

bindrcpp

0.2.2

bit

1.1-14

bit64

0.9-7

bitops

1.0-6

blob

1.1.1

boot

1.3-20

brew

1.0-6

broom

0.5.0

callr

3.0.0

car

3.0-2

carData

3.0-2

caret

6.0-80

cellranger

1.1.0

chron

2.3-53

class

7.3-14

cli

1.0.1

clipr

0.4.1

clisymbols

1.2.0

cluster

2.0.7-1

codetools

0.2-15

colorspace

1.3-2

commonmark

1.6

compiler

3.4.4

crayon

1.3.4

curl

3.2

CVST

0.2-2

data.table

1.11.8

datasets

3.4.4

DBI

1.0.0

ddalpha

1.3.4

DEoptimR

1.0-8

desc

1.2.0

devtools

2.0.1

digest

0.6.18

dimRed

0.1.0

doMC

1.3.5

dplyr

0.7.7

DRR

0.0.3

fansi

0.4.0

forcats

0.3.0

foreach

1.4.4

foreign

0.8-70

fs

1.2.6

gbm

2.1.4

geometry

0.3-6

ggplot2

3.1.0

gh

1.0.1

git2r

0.23.0

glmnet

2.0-16

glue

1.3.0

gower

0.1.2

graphics

3.4.4

grDevices

3.4.4

grid

3.4.4

gridExtra

2.3

gsubfn

0.7

gtable

0.2.0

h2o

3.20.0.8

haven

1.1.2

hms

0.4.2

httr

1.3.1

hwriter

1.3.2

hwriterPlus

1.0-3

ini

0.3.1

ipred

0.9-8

iterators

1.0.10

jsonlite

1.5

kernlab

0.9-27

KernSmooth

2.23-15

labeling

0.3

lattice

0.20-38

lava

1.6.3

lazyeval

0.2.1

littler

0.3.5

lme4

1.1-18-1

lubridate

1.7.4

magic

1.5-9

magrittr

1.5

mapproj

1.2.6

maps

3.3.0

maptools

0.9-4

MASS

7.3-50

Matrix

1.2-14

MatrixModels

0.4-1

memoise

1.1.0

methods

3.4.4

mgcv

1.8-25

mime

0.6

minqa

1.2.4

ModelMetrics

1.2.2

munsell

0.5.0

mvtnorm

1.0-8

nlme

3.1-137

nloptr

1.2.1

nnet

7.3-12

numDeriv

2016.8-1

openssl

1.0.2

openxlsx

4.1.0

parallel

3.4.4

pbkrtest

0.4-7

pillar

1.3.0

pkgbuild

1.0.2

pkgconfig

2.0.2

pkgKitten

0.1.4

pkgload

1.0.2

plogr

0.2.0

pls

2.7-0

plyr

1.8.4

praise

1.0.0

prettyunits

1.0.2

pROC

1.13.0

processx

3.2.0

prodlim

2018.04.18

proto

1.0.0

ps

1.2.0

purrr

0.2.5

quantreg

5.36

R.methodsS3

1.7.1

R.oo

1.22.0

R.utils

2.7.0

R6

2.3.0

randomForest

4.6-14

rcmdcheck

1.3.1

RColorBrewer

1.1-2

Rcpp

0.12.19

RcppEigen

0.3.3.4.0

RcppRoll

0.3.0

RCurl

1.95-4.11

readr

1.1.1

readxl

1.1.0

recipes

0.1.3

rematch

1.0.1

remotes

2.0.2

reshape2

1.4.3

rio

0.5.10

rlang

0.3.0.1

robustbase

0.93-3

RODBC

1.3-15

roxygen2

6.1.0

rpart

4.1-13

rprojroot

1.3-2

Rserve

1.7-3

RSQLite

2.1.1

rstudioapi

0.8

scales

1.0.0

sessioninfo

1.1.1

sfsmisc

1.1-2

sp

1.3-1

SparkR

2.4.0

SparseM

1.77

spatial

7.3-11

splines

3.4.4

sqldf

0.4-11

SQUAREM

2017.10-1

statmod

1.4.30

stats

3.4.4

stats4

3.4.4

stringi

1.2.4

stringr

1.3.1

survival

2.43-1

tcltk

3.4.4

TeachingDemos

2.10

testthat

2.0.1

tibble

1.4.2

tidyr

0.8.2

tidyselect

0.2.5

timeDate

3043.102

tools

3.4.4

usethis

1.4.0

utf8

1.1.4

utils

3.4.4

viridisLite

0.3.0

whisker

0.3-2

withr

2.1.2

xml2

1.2.0

xopen

1.0.0

zip

1.0.0

Installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.8.10

com.amazonaws

aws-java-sdk-autoscaling

1.11.313

com.amazonaws

aws-java-sdk-cloudformation

1.11.313

com.amazonaws

aws-java-sdk-cloudfront

1.11.313

com.amazonaws

aws-java-sdk-cloudhsm

1.11.313

com.amazonaws

aws-java-sdk-cloudsearch

1.11.313

com.amazonaws

aws-java-sdk-cloudtrail

1.11.313

com.amazonaws

aws-java-sdk-cloudwatch

1.11.313

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.313

com.amazonaws

aws-java-sdk-codedeploy

1.11.313

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.313

com.amazonaws

aws-java-sdk-cognitosync

1.11.313

com.amazonaws

aws-java-sdk-config

1.11.313

com.amazonaws

aws-java-sdk-core

1.11.313

com.amazonaws

aws-java-sdk-datapipeline

1.11.313

com.amazonaws

aws-java-sdk-directconnect

1.11.313

com.amazonaws

aws-java-sdk-directory

1.11.313

com.amazonaws

aws-java-sdk-dynamodb

1.11.313

com.amazonaws

aws-java-sdk-ec2

1.11.313

com.amazonaws

aws-java-sdk-ecs

1.11.313

com.amazonaws

aws-java-sdk-efs

1.11.313

com.amazonaws

aws-java-sdk-elasticache

1.11.313

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.313

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.313

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.313

com.amazonaws

aws-java-sdk-emr

1.11.313

com.amazonaws

aws-java-sdk-glacier

1.11.313

com.amazonaws

aws-java-sdk-iam

1.11.313

com.amazonaws

aws-java-sdk-importexport

1.11.313

com.amazonaws

aws-java-sdk-kinesis

1.11.313

com.amazonaws

aws-java-sdk-kms

1.11.313

com.amazonaws

aws-java-sdk-lambda

1.11.313

com.amazonaws

aws-java-sdk-logs

1.11.313

com.amazonaws

aws-java-sdk-machinelearning

1.11.313

com.amazonaws

aws-java-sdk-opsworks

1.11.313

com.amazonaws

aws-java-sdk-rds

1.11.313

com.amazonaws

aws-java-sdk-redshift

1.11.313

com.amazonaws

aws-java-sdk-route53

1.11.313

com.amazonaws

aws-java-sdk-s3

1.11.313

com.amazonaws

aws-java-sdk-ses

1.11.313

com.amazonaws

aws-java-sdk-simpledb

1.11.313

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.313

com.amazonaws

aws-java-sdk-sns

1.11.313

com.amazonaws

aws-java-sdk-sqs

1.11.313

com.amazonaws

aws-java-sdk-ssm

1.11.313

com.amazonaws

aws-java-sdk-storagegateway

1.11.313

com.amazonaws

aws-java-sdk-sts

1.11.313

com.amazonaws

aws-java-sdk-support

1.11.313

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.313

com.amazonaws

jmespath-java

1.11.313

com.carrotsearch

hppc

0.7.2

com.chuusai

shapeless_2.11

2.3.2

com.clearspring.analytics

stream

2.7.0

com.databricks

Rserve

1.8-3

com.databricks

dbml-local_2.11

0.5.0-db8-spark2.4

com.databricks

dbml-local_2.11-tests

0.5.0-db8-spark2.4

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.11

0.4.15-9

com.databricks.scalapb

scalapb-runtime_2.11

0.4.15-9

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.0.0

com.fasterxml.jackson.core

jackson-annotations

2.6.7

com.fasterxml.jackson.core

jackson-core

2.6.7

com.fasterxml.jackson.core

jackson-databind

2.6.7.1

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.6.7

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.6.7

com.fasterxml.jackson.module

jackson-module-paranamer

2.6.7

com.fasterxml.jackson.module

jackson-module-scala_2.11

2.6.7.1

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.3.2-2

com.github.rwl

jtransforms

2.4.0

com.google.code.findbugs

jsr305

2.0.1

com.google.code.gson

gson

2.2.4

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.googlecode.javaewah

JavaEWAH

0.3.2

com.h2database

h2

1.3.174

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.mchange

c3p0

0.9.5.1

com.mchange

mchange-commons-java

0.2.10

com.microsoft.azure

azure-data-lake-store-sdk

2.2.8

com.microsoft.sqlserver

mssql-jdbc

6.2.2.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.11

0.3

com.twitter

chill-java

0.9.3

com.twitter

chill_2.11

0.9.3

com.twitter

parquet-hadoop-bundle

1.6.0

com.twitter

util-app_2.11

6.23.0

com.twitter

util-core_2.11

6.23.0

com.twitter

util-jvm_2.11

6.23.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging-api_2.11

2.1.2

com.typesafe.scala-logging

scala-logging-slf4j_2.11

2.1.2

com.univocity

univocity-parsers

2.7.3

com.vlkan

flatbuffers

1.2.0-3f79e055

com.zaxxer

HikariCP

3.1.0

commons-beanutils

commons-beanutils

1.7.0

commons-beanutils

commons-beanutils-core

1.8.0

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

3.1

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.7

io.airlift

aircompressor

0.10

io.dropwizard.metrics

metrics-core

3.1.5

io.dropwizard.metrics

metrics-ganglia

3.1.5

io.dropwizard.metrics

metrics-graphite

3.1.5

io.dropwizard.metrics

metrics-healthchecks

3.1.5

io.dropwizard.metrics

metrics-jetty9

3.1.5

io.dropwizard.metrics

metrics-json

3.1.5

io.dropwizard.metrics

metrics-jvm

3.1.5

io.dropwizard.metrics

metrics-log4j

3.1.5

io.dropwizard.metrics

metrics-servlets

3.1.5

io.netty

netty

3.9.9.Final

io.netty

netty-all

4.1.17.Final

io.prometheus

simpleclient

0.0.16

io.prometheus

simpleclient_common

0.0.16

io.prometheus

simpleclient_dropwizard

0.0.16

io.prometheus

simpleclient_servlet

0.0.16

io.prometheus.jmx

collector

0.7

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.validation

validation-api

1.1.0.Final

javax.ws.rs

javax.ws.rs-api

2.0.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.11

joda-time

joda-time

2.9.3

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

net.hydromatic

eigenbase-properties

1.1.5

net.razorvine

pyrolite

4.13

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.5

net.snowflake

snowflake-jdbc

3.6.15

net.snowflake

spark-snowflake_2.11

2.4.10-spark_2.4

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt

oncrpc

1.0.7

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.4

org.antlr

antlr4-runtime

4.7

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

0.10.0

org.apache.arrow

arrow-memory

0.10.0

org.apache.arrow

arrow-vector

0.10.0

org.apache.avro

avro

1.8.2

org.apache.avro

avro-ipc

1.8.2

org.apache.avro

avro-mapred-hadoop2

1.8.2

org.apache.calcite

calcite-avatica

1.2.0-incubating

org.apache.calcite

calcite-core

1.2.0-incubating

org.apache.calcite

calcite-linq4j

1.2.0-incubating

org.apache.commons

commons-compress

1.8.1

org.apache.commons

commons-crypto

1.0.0

org.apache.commons

commons-lang3

3.5

org.apache.commons

commons-math3

3.4.1

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.3

org.apache.hadoop

hadoop-auth

2.7.3

org.apache.hadoop

hadoop-client

2.7.3

org.apache.hadoop

hadoop-common

2.7.3

org.apache.hadoop

hadoop-hdfs

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.3

org.apache.hadoop

hadoop-yarn-api

2.7.3

org.apache.hadoop

hadoop-yarn-client

2.7.3

org.apache.hadoop

hadoop-yarn-common

2.7.3

org.apache.hadoop

hadoop-yarn-server-common

2.7.3

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.4

org.apache.httpcomponents

httpcore

4.4.8

org.apache.ivy

ivy

2.4.0

org.apache.orc

orc-core-nohive

1.5.2

org.apache.orc

orc-mapreduce-nohive

1.5.2

org.apache.orc

orc-shims

1.5.2

org.apache.parquet

parquet-column

1.10.1-databricks3

org.apache.parquet

parquet-common

1.10.1-databricks3

org.apache.parquet

parquet-encoding

1.10.1-databricks3

org.apache.parquet

parquet-format

2.4.0

org.apache.parquet

parquet-hadoop

1.10.1-databricks3

org.apache.parquet

parquet-jackson

1.10.1-databricks3

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.9.3

org.apache.xbean

xbean-asm6-shaded

4.8

org.apache.zookeeper

zookeeper

3.4.6

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.9

org.codehaus.janino

janino

3.0.9

org.datanucleus

datanucleus-api-jdo

3.2.6

org.datanucleus

datanucleus-core

3.2.10

org.datanucleus

datanucleus-rdbms

3.2.9

org.eclipse.jetty

jetty-client

9.3.20.v20170531

org.eclipse.jetty

jetty-continuation

9.3.20.v20170531

org.eclipse.jetty

jetty-http

9.3.20.v20170531

org.eclipse.jetty

jetty-io

9.3.20.v20170531

org.eclipse.jetty

jetty-jndi

9.3.20.v20170531

org.eclipse.jetty

jetty-plus

9.3.20.v20170531

org.eclipse.jetty

jetty-proxy

9.3.20.v20170531

org.eclipse.jetty

jetty-security

9.3.20.v20170531

org.eclipse.jetty

jetty-server

9.3.20.v20170531

org.eclipse.jetty

jetty-servlet

9.3.20.v20170531

org.eclipse.jetty

jetty-servlets

9.3.20.v20170531

org.eclipse.jetty

jetty-util

9.3.20.v20170531

org.eclipse.jetty

jetty-webapp

9.3.20.v20170531

org.eclipse.jetty

jetty-xml

9.3.20.v20170531

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.4.0-b34

org.glassfish.hk2

hk2-locator

2.4.0-b34

org.glassfish.hk2

hk2-utils

2.4.0-b34

org.glassfish.hk2

osgi-resource-locator

1.0.1

org.glassfish.hk2.external

aopalliance-repackaged

2.4.0-b34

org.glassfish.hk2.external

javax.inject

2.4.0-b34

org.glassfish.jersey.bundles.repackaged

jersey-guava

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet-core

2.22.2

org.glassfish.jersey.core

jersey-client

2.22.2

org.glassfish.jersey.core

jersey-common

2.22.2

org.glassfish.jersey.core

jersey-server

2.22.2

org.glassfish.jersey.media

jersey-media-jaxb

2.22.2

org.hibernate

hibernate-validator

5.1.1.Final

org.iq80.snappy

snappy

0.2

org.javassist

javassist

3.18.1-GA

org.jboss.logging

jboss-logging

3.1.3.GA

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.11

3.5.3

org.json4s

json4s-core_2.11

3.5.3

org.json4s

json4s-jackson_2.11

3.5.3

org.json4s

json4s-scalap_2.11

3.5.3

org.lz4

lz4-java

1.4.0

org.mariadb.jdbc

mariadb-java-client

2.1.2

org.mockito

mockito-all

1.9.5

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.5.11

org.rocksdb

rocksdbjni

5.2.1

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.11

2.11.8

org.scala-lang

scala-library_2.11

2.11.8

org.scala-lang

scala-reflect_2.11

2.11.8

org.scala-lang.modules

scala-parser-combinators_2.11

1.0.2

org.scala-lang.modules

scala-xml_2.11

1.0.5

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.11

1.12.5

org.scalactic

scalactic_2.11

3.0.3

org.scalanlp

breeze-macros_2.11

0.13.2

org.scalanlp

breeze_2.11

0.13.2

org.scalatest

scalatest_2.11

3.0.3

org.slf4j

jcl-over-slf4j

1.7.16

org.slf4j

jul-to-slf4j

1.7.16

org.slf4j

slf4j-api

1.7.16

org.slf4j

slf4j-log4j12

1.7.16

org.spark-project.hive

hive-beeline

1.2.1.spark2

org.spark-project.hive

hive-cli

1.2.1.spark2

org.spark-project.hive

hive-exec

1.2.1.spark2

org.spark-project.hive

hive-jdbc

1.2.1.spark2

org.spark-project.hive

hive-metastore

1.2.1.spark2

org.spark-project.spark

unused

1.0.0

org.spire-math

spire-macros_2.11

0.13.0

org.spire-math

spire_2.11

0.13.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.tukaani

xz

1.5

org.typelevel

machinist_2.11

0.6.1

org.typelevel

macro-compat_2.11

1.1.1

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.7.1

org.yaml

snakeyaml

1.16

oro

oro

2.0.8

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52