Databricks Runtime 4.0 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in March 2018.

Important

This release was deprecated on November 1, 2018. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks support lifecycles.

The following release notes provide information about Databricks Runtime 4.0, powered by Apache Spark.

Changes and improvements

  • The JSON data source now tries to auto-detect encoding instead of assuming it to be UTF-8. In cases where the auto-detection fails, users can specify the charset option to enforce a certain encoding. See Charset auto-detection.

  • Scoring and prediction using Spark MLlib pipelines in Structured Streaming is fully supported.

  • Databricks ML Model Export is fully supported. With this feature, you can train a Spark MLlib model on Databricks, export it with a function call, and use a Databricks library in the system of your choice to import the model and score new data.

  • A new Spark data source implementation offers scalable read/write access to Azure Synapse Analytics. See Spark - Synapse Analytics Connector.

  • The schema of the from_json function is now always converted to a nullable one. In other words, all fields, including nested ones, are nullable. This ensures that the data is compatible with the schema, preventing corruption after writing the data to parquet when a field is missing in the data and the user-provided schema declares the field as non-nullable.

  • Upgraded some installed Python libraries:

    • futures: from 3.1.1 to 3.2.0

    • pandas: from 0.18.1 to 0.19.2

    • pyarrow: from 0.4.1 to 0.8.0

    • setuptools: from 38.2.3 to 38.5.1

    • tornado: 4.5.2 to 4.5.3

  • Upgraded several installed R libraries. See Installed R Libraries.

  • Upgraded AWS Java SDK from 1.11.126 to 1.11.253.

  • Upgraded SQL Server JDBC driver from 6.1.0.jre8 to 6.2.2.jre8.

  • Upgraded PostgreSQL JDBC driver from 9.4-1204-jdbc41 to 42.1.4.

Apache Spark

Databricks Runtime 4.0 includes Apache Spark 2.3.0.

Core, PySpark, and Spark SQL

Major features

  • Vectorized ORC Reader: [SPARK-16060]: Adds support for new ORC reader that substantially improves the ORC scan throughput through vectorization (2-5x). To enable the reader, users can set spark.sql.orc.impl to native.

  • Spark History Server V2: [SPARK-18085]: A new spark history server (SHS) backend that provides better scalability for large-scale applications with a more efficient event storage mechanism.

  • Data source API V2: [SPARK-15689][SPARK-22386]: An experimental API for plugging in new data sources in Spark. The new API attempts to address several limitations of the V1 API and aims to facilitate development of highly-performant, easy-to-maintain, and extensible external data sources. This API is still undergoing active development and breaking changes should be expected.

  • PySpark Performance Enhancements: [SPARK-22216][SPARK-21187]: Significant improvements in Python performance and interoperability by fast data serialization and vectorized execution.

Performance and stability

Other notable changes

Structured Streaming

Continuous Processing

  • A new execution engine that can execute streaming queries with sub-millisecond end-to-end latency by changing only a single line of user code. To learn more see the programming guide.

Stream-Stream Joins

  • Ability to join two streams of data, buffering rows until matching tuples arrive in the other stream. Predicates can be used against event time columns to bound the amount of state that needs to be retained.

Streaming API V2

  • An experimental API for plugging in new source and sinks that works for batch, micro-batch, and continuous execution. This API is still undergoing active development, and breaking changes should be expected.

MLlib

Highlights

  • ML Prediction now works with Structured Streaming, using updated APIs. Details follow.

New and improved APIs

  • [SPARK-21866]: Built-in support for reading images into a DataFrame (Scala/Java/Python).

  • [SPARK-19634]: DataFrame functions for descriptive summary statistics over vector columns (Scala/Java).

  • [SPARK-14516]: ClusteringEvaluator for tuning clustering algorithms, supporting Cosine silhouette and squared Euclidean silhouette metrics (Scala/Java/Python).

  • [SPARK-3181]: Robust linear regression with Huber loss (Scala/Java/Python).

  • [SPARK-13969]: FeatureHasher transformer (Scala/Java/Python).

  • Multiple column support for several feature transformers:

  • [SPARK-21633] and SPARK-21542]: Improved support for custom pipeline components in Python.

New features

  • [SPARK-21087]: CrossValidator and TrainValidationSplit can collect all models when fitting (Scala/Java). This allows you to inspect or save all fitted models.

  • [SPARK-19357]: Meta-algorithms CrossValidator, TrainValidationSplit,OneVsRest support a parallelism Param for fitting multiple sub-models in parallel Spark jobs.

  • [SPARK-17139]: Model summary for multinomial logistic regression (Scala/Java/Python)

  • [SPARK-18710]: Add offset in GLM.

  • [SPARK-20199]: Added featureSubsetStrategy Param to GBTClassifier and GBTRegressor. Using this to subsample features can significantly improve training speed; this option has been a key strength of xgboost.

Other notable changes

  • [SPARK-22156]: Fixed Word2Vec learning rate scaling with num iterations. The new learning rate is set to match the original Word2Vec C code and should give better results from training.

  • [SPARK-22289]: Add JSON support for Matrix parameters (This fixed a bug for ML persistence with LogisticRegressionModel when using bounds on coefficients.)

  • [SPARK-22700]: Bucketizer.transform incorrectly drops row containing NaN. When Param handleInvalid was set to “skip,” Bucketizer would drop a row with a valid value in the input column if another (irrelevant) column had a NaN value.

  • [SPARK-22446]: Catalyst optimizer sometimes caused StringIndexerModel to throw an incorrect “Unseen label” exception when handleInvalid was set to “error.” This could happen for filtered data, due to predicate push-down, causing errors even after invalid rows had already been filtered from the input dataset.

  • [SPARK-21681]: Fixed an edge case bug in multinomial logistic regression that resulted in incorrect coefficients when some features had zero variance.

  • Major optimizations:

    • [SPARK-22707]: Reduced memory consumption for CrossValidator.

    • [SPARK-22949]: Reduced memory consumption for TrainValidationSplit.

    • [SPARK-21690]: Imputer should train using a single pass over the data.

    • [SPARK-14371]: OnlineLDAOptimizer avoids collecting statistics to the driver for each mini-batch.

SparkR

The main focus of SparkR in the 2.3.0 release was improving the stability of UDFs and adding several new SparkR wrappers around existing APIs:

Major features

GraphX

Optimizations

  • [SPARK-5484]: Pregel now checkpoints periodically to avoid StackOverflowErrors.

  • [SPARK-21491]: Small performance improvement in several places.

Deprecations

Python

  • [SPARK-23122]: Deprecate register* for UDFs in SQLContext and Catalog in PySpark

MLlib

  • [SPARK-13030]: OneHotEncoder has been deprecated and will be removed in 3.0. It has been replaced by the new OneHotEncoderEstimator. OneHotEncoderEstimator will be renamed to OneHotEncoder in 3.0 (but OneHotEncoderEstimator will be kept as an alias).

Changes of behavior

SparkSQL

  • [SPARK-22036]: By default arithmetic operations between decimals return a rounded value if an exact representation is not possible (instead of returning NULL in the prior versions)

  • [SPARK-22937]: When all inputs are binary, SQL elt() returns an output as binary. Otherwise, it returns as a string. In prior versions, it always returned as a string regardless of input types.

  • [SPARK-22895]: The Join/Filter’s deterministic predicates that are after the first non-deterministic predicates are also pushed down/through the child operators, if possible. In the prior versions, these filters were not eligible for predicate pushdown.

  • [SPARK-22771]: When all inputs are binary, functions.concat() returns an output as binary. Otherwise, it returns as a string. In the prior versions, it always returned as a string regardless of input types.

  • [SPARK-22489]: When either of the join sides is broadcastable, we prefer to broadcast the table that is explicitly specified in a broadcast hint.

  • [SPARK-22165]: Partition column inference previously found incorrect common type for different inferred types. For example, previously it ended up with double type as the common type for double type and date type. Now it finds the correct common type for such conflicts. For details, see the migration guide.

  • [SPARK-22100]: The percentile_approx function previously accepted numeric type input and outputted double type results. Now it supports date type, timestamp type and numeric types as input types. The result type is also changed to be the same as the input type, which is more reasonable for percentiles.

  • [SPARK-21610]: the queries from raw JSON/CSV files are disallowed when the referenced columns only include the internal corrupt record column (named _corrupt_record by default). Instead, you can cache or save the parsed results and then send the same query.

  • [SPARK-23421]: Since Spark 2.2.1 and 2.3.0, the schema is always inferred at runtime when the data source tables have the columns that exist in both partition schema and data schema. The inferred schema does not have the partitioned columns. When reading the table, Spark respects the partition values of these overlapping columns instead of the values stored in the data source files. In 2.2.0 and 2.1.x release, the inferred schema is partitioned but the data of the table is invisible to users (i.e., the result set is empty).

PySpark

  • [SPARK-19732]: na.fill() or fillna also accepts boolean and replaces nulls with booleans. In prior Spark versions, PySpark just ignores it and returns the original Dataset/DataFrame.

  • [SPARK-22395]: pandas 0.19.2 or upper is required for using pandas related functionalities, such as toPandas, createDataFrame from pandas DataFrame, etc.

  • [SPARK-22395]: The behavior of timestamp values for pandas related functionalities was changed to respect session timezone, which is ignored in the prior versions.

  • [SPARK-23328]: df.replace does not allow to omit value when to_replace is not a dictionary. Previously, value could be omitted in the other cases and had None by default, which is counter-intuitive and error prone.

MLlib

  • Breaking API Changes: The class and trait hierarchy for logistic regression model summaries was changed to be cleaner and better accommodate the addition of the multi-class summary. This is a breaking change for user code that casts a LogisticRegressionTrainingSummary to a BinaryLogisticRegressionTrainingSummary. Users should instead use the model.binarySummary method. See [SPARK-17139]: for more detail (note this is an @Experimental API). This does not affect the Python summary method, which will still work correctly for both multinomial and binary cases.

  • [SPARK-21806]: BinaryClassificationMetrics.pr(): first point (0.0, 1.0) is misleading and has been replaced by (0.0, p) where precision p matches the lowest recall point.

  • [SPARK-16957]: Decision trees now use weighted midpoints when choosing split values. This may change results from model training.

  • [SPARK-14657]: RFormula without an intercept now outputs the reference category when encoding string terms, in order to match native R behavior. This may change results from model training.

  • [SPARK-21027]: The default parallelism used in OneVsRest is now set to 1 (i.e. serial). In 2.2 and earlier versions, the level of parallelism was set to the default threadpool size in Scala. This may change performance.

  • [SPARK-21523]: Upgraded Breeze to 0.13.2. This included an important bug fix in strong Wolfe line search for L-BFGS.

  • [SPARK-15526]: The JPMML dependency is now shaded.

  • Also see the “Bug fixes” section for behavior changes resulting from fixing bugs.

Known issues

  • [SPARK-23523][SQL]: Incorrect result caused by the rule OptimizeMetadataOnlyQuery.

  • [SPARK-23406]: Bugs in stream-stream self-joins.

System environment

  • Operating System: Ubuntu 16.04.4 LTS

  • Java: 1.8.0_151

  • Scala: 2.11.8

  • Python: 2.7.12 (or 3.5.2 if using Python 3)

  • R: R version 3.4.3 (2017-11-30)

  • GPU clusters: The following NVIDIA GPU libraries are installed:

    • Tesla driver 375.66

    • CUDA 8.0

    • CUDNN 6.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

ansi2html

1.1.1

argparse

1.2.1

backports-abc

0.5

boto

2.42.0

boto3

1.4.1

botocore

1.4.70

brewer2mpl

1.4.1

certifi

2016.2.28

cffi

1.7.0

chardet

2.3.0

colorama

0.3.7

configobj

5.0.6

cryptography

1.5

cycler

0.10.0

Cython

0.24.1

decorator

4.0.10

docutils

0.14

enum34

1.1.6

et-xmlfile

1.0.1

freetype-py

1.0.2

funcsigs

1.0.2

fusepy

2.0.4

futures

3.2.0

ggplot

0.6.8

html5lib

0.999

idna

2.1

ipaddress

1.0.16

ipython

2.2.0

ipython-genutils

0.1.0

jdcal

1.2

Jinja2

2.8

jmespath

0.9.0

llvmlite

0.13.0

lxml

3.6.4

MarkupSafe

0.23

matplotlib

1.5.3

mpld3

0.2

msgpack-python

0.4.7

ndg-httpsclient

0.3.3

numba

0.28.1

numpy

1.11.1

openpyxl

2.3.2

pandas

0.19.2

pathlib2

2.1.0

patsy

0.4.1

pexpect

4.0.1

pickleshare

0.7.4

Pillow

3.3.1

pip

9.0.1

ply

3.9

prompt-toolkit

1.0.7

psycopg2

2.6.2

ptyprocess

0.5.1

py4j

0.10.3

pyarrow

0.8.0

pyasn1

0.1.9

pycparser

2.14

Pygments

2.1.3

PyGObject

3.20.0

pyOpenSSL

16.0.0

pyparsing

2.2.0

pypng

0.0.18

Python

2.7.12

python-dateutil

2.5.3

python-geohash

0.8.5

pytz

2016.6.1

requests

2.11.1

s3transfer

0.1.9

scikit-learn

0.18.1

scipy

0.18.1

scour

0.32

seaborn

0.7.1

setuptools

38.5.1

simplejson

3.8.2

simples3

1.0

singledispatch

3.4.0.3

six

1.10.0

statsmodels

0.6.1

tornado

4.5.3

traitlets

4.3.0

urllib3

1.19.1

virtualenv

15.0.1

wcwidth

0.1.7

wheel

0.30.0

wsgiref

0.1.2

Installed R libraries

Library

Version

Library

Version

Library

Version

abind

1.4-5

assertthat

0.2.0

backports

1.1.1

base

3.4.3

BH

1.65.0-1

bindr

0.1

bindrcpp

0.2

bit

1.1-12

bit64

0.9-7

bitops

1.0-6

blob

1.1.0

boot

1.3-20

brew

1.0-6

broom

0.4.3

car

2.1-6

caret

6.0-77

chron

2.3-51

class

7.3-14

cluster

2.0.6

codetools

0.2-15

colorspace

1.3-2

commonmark

1.4

compiler

3.4.3

crayon

1.3.4

curl

3.0

CVST

0.2-1

data.table

1.10.4-3

datasets

3.4.3

DBI

0.7

ddalpha

1.3.1

DEoptimR

1.0-8

desc

1.1.1

devtools

1.13.4

dichromat

2.0-0

digest

0.6.12

dimRed

0.1.0

doMC

1.3.4

dplyr

0.7.4

DRR

0.0.2

foreach

1.4.3

foreign

0.8-69

gbm

2.1.3

ggplot2

2.2.1

git2r

0.19.0

glmnet

2.0-13

glue

1.2.0

gower

0.1.2

graphics

3.4.3

grDevices

3.4.3

grid

3.4.3

gsubfn

0.6-6

gtable

0.2.0

h2o

3.16.0.1

httr

1.3.1

hwriter

1.3.2

hwriterPlus

1.0-3

ipred

0.9-6

iterators

1.0.8

jsonlite

1.5

kernlab

0.9-25

KernSmooth

2.23-15

labeling

0.3

lattice

0.20-35

lava

1.5.1

lazyeval

0.2.1

littler

0.3.2

lme4

1.1-14

lubridate

1.7.1

magrittr

1.5

mapproj

1.2-5

maps

3.2.0

MASS

7.3-48

Matrix

1.2-11

MatrixModels

0.4-1

memoise

1.1.0

methods

3.4.3

mgcv

1.8-23

mime

0.5

minqa

1.2.4

mnormt

1.5-5

ModelMetrics

1.1.0

munsell

0.4.3

mvtnorm

1.0-6

nlme

3.1-131

nloptr

1.0.4

nnet

7.3-12

numDeriv

2016.8-1

openssl

0.9.9

parallel

3.4.3

pbkrtest

0.4-7

pkgconfig

2.0.1

pkgKitten

0.1.4

plogr

0.1-1

plyr

1.8.4

praise

1.0.0

pROC

1.10.0

prodlim

1.6.1

proto

1.0.0

psych

1.7.8

purrr

0.2.4

quantreg

5.34

R.methodsS3

1.7.1

R.oo

1.21.0

R.utils

2.6.0

R6

2.2.2

randomForest

4.6-12

RColorBrewer

1.1-2

Rcpp

0.12.14

RcppEigen

0.3.3.3.1

RcppRoll

0.2.2

RCurl

1.95-4.8

recipes

0.1.1

reshape2

1.4.2

rlang

0.1.4

robustbase

0.92-8

RODBC

1.3-15

roxygen2

6.0.1

rpart

4.1-12

rprojroot

1.2

Rserve

1.7-3

RSQLite

2.0

rstudioapi

0.7

scales

0.5.0

sfsmisc

1.1-1

sp

1.2-5

SparkR

2.3.0

SparseM

1.77

spatial

7.3-11

splines

3.4.3

sqldf

0.4-11

statmod

1.4.30

stats

3.4.3

stats4

3.4.3

stringi

1.1.6

stringr

1.2.0

survival

2.41-3

tcltk

3.4.3

TeachingDemos

2.10

testthat

1.0.2

tibble

1.3.4

tidyr

0.7.2

tidyselect

0.2.3

timeDate

3042.101

tools

3.4.3

utils

3.4.3

viridisLite

0.2.0

whisker

0.3-2

withr

2.1.0

xml2

1.1.1

Installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.7.3

com.amazonaws

aws-java-sdk-autoscaling

1.11.253

com.amazonaws

aws-java-sdk-cloudformation

1.11.253

com.amazonaws

aws-java-sdk-cloudfront

1.11.253

com.amazonaws

aws-java-sdk-cloudhsm

1.11.253

com.amazonaws

aws-java-sdk-cloudsearch

1.11.253

com.amazonaws

aws-java-sdk-cloudtrail

1.11.253

com.amazonaws

aws-java-sdk-cloudwatch

1.11.253

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.11.253

com.amazonaws

aws-java-sdk-codedeploy

1.11.253

com.amazonaws

aws-java-sdk-cognitoidentity

1.11.253

com.amazonaws

aws-java-sdk-cognitosync

1.11.253

com.amazonaws

aws-java-sdk-config

1.11.253

com.amazonaws

aws-java-sdk-core

1.11.253

com.amazonaws

aws-java-sdk-datapipeline

1.11.253

com.amazonaws

aws-java-sdk-directconnect

1.11.253

com.amazonaws

aws-java-sdk-directory

1.11.253

com.amazonaws

aws-java-sdk-dynamodb

1.11.253

com.amazonaws

aws-java-sdk-ec2

1.11.253

com.amazonaws

aws-java-sdk-ecs

1.11.253

com.amazonaws

aws-java-sdk-efs

1.11.253

com.amazonaws

aws-java-sdk-elasticache

1.11.253

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.11.253

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.11.253

com.amazonaws

aws-java-sdk-elastictranscoder

1.11.253

com.amazonaws

aws-java-sdk-emr

1.11.253

com.amazonaws

aws-java-sdk-glacier

1.11.253

com.amazonaws

aws-java-sdk-iam

1.11.253

com.amazonaws

aws-java-sdk-importexport

1.11.253

com.amazonaws

aws-java-sdk-kinesis

1.11.253

com.amazonaws

aws-java-sdk-kms

1.11.253

com.amazonaws

aws-java-sdk-lambda

1.11.253

com.amazonaws

aws-java-sdk-logs

1.11.253

com.amazonaws

aws-java-sdk-machinelearning

1.11.253

com.amazonaws

aws-java-sdk-opsworks

1.11.253

com.amazonaws

aws-java-sdk-rds

1.11.253

com.amazonaws

aws-java-sdk-redshift

1.11.253

com.amazonaws

aws-java-sdk-route53

1.11.253

com.amazonaws

aws-java-sdk-s3

1.11.253

com.amazonaws

aws-java-sdk-ses

1.11.253

com.amazonaws

aws-java-sdk-simpledb

1.11.253

com.amazonaws

aws-java-sdk-simpleworkflow

1.11.253

com.amazonaws

aws-java-sdk-sns

1.11.253

com.amazonaws

aws-java-sdk-sqs

1.11.253

com.amazonaws

aws-java-sdk-ssm

1.11.253

com.amazonaws

aws-java-sdk-storagegateway

1.11.253

com.amazonaws

aws-java-sdk-sts

1.11.253

com.amazonaws

aws-java-sdk-support

1.11.253

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.11.253

com.amazonaws

jmespath-java

1.11.253

com.carrotsearch

hppc

0.7.2

com.chuusai

shapeless_2.11

2.3.2

com.clearspring.analytics

stream

2.7.0

com.databricks

Rserve

1.8-3

com.databricks

dbml-local_2.11

0.3.0-db1-spark2.3

com.databricks

dbml-local_2.11-tests

0.3.0-db1-spark2.3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.11

0.4.15-9

com.databricks.scalapb

scalapb-runtime_2.11

0.4.15-9

com.esotericsoftware

kryo-shaded

3.0.3

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.0.0

com.fasterxml.jackson.core

jackson-annotations

2.6.7

com.fasterxml.jackson.core

jackson-core

2.6.7

com.fasterxml.jackson.core

jackson-databind

2.6.7.1

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.6.7

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.6.7

com.fasterxml.jackson.module

jackson-module-paranamer

2.6.7

com.fasterxml.jackson.module

jackson-module-scala_2.11

2.6.7.1

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.3.2-2

com.github.rwl

jtransforms

2.4.0

com.google.code.findbugs

jsr305

2.0.1

com.google.code.gson

gson

2.2.4

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.googlecode.javaewah

JavaEWAH

0.3.2

com.h2database

h2

1.3.174

com.jamesmurty.utils

java-xmlbuilder

1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.mchange

c3p0

0.9.5.1

com.mchange

mchange-commons-java

0.2.10

com.microsoft.azure

azure-data-lake-store-sdk

2.0.11

com.microsoft.sqlserver

mssql-jdbc

6.2.2.jre8

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.11

0.3

com.twitter

chill-java

0.8.4

com.twitter

chill_2.11

0.8.4

com.twitter

parquet-hadoop-bundle

1.6.0

com.twitter

util-app_2.11

6.23.0

com.twitter

util-core_2.11

6.23.0

com.twitter

util-jvm_2.11

6.23.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging-api_2.11

2.1.2

com.typesafe.scala-logging

scala-logging-slf4j_2.11

2.1.2

com.univocity

univocity-parsers

2.5.9

com.vlkan

flatbuffers

1.2.0-3f79e055

com.zaxxer

HikariCP

2.4.1

commons-beanutils

commons-beanutils

1.7.0

commons-beanutils

commons-beanutils-core

1.8.0

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

2.2

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.7

io.airlift

aircompressor

0.8

io.dropwizard.metrics

metrics-core

3.1.5

io.dropwizard.metrics

metrics-ganglia

3.1.5

io.dropwizard.metrics

metrics-graphite

3.1.5

io.dropwizard.metrics

metrics-healthchecks

3.1.5

io.dropwizard.metrics

metrics-jetty9

3.1.5

io.dropwizard.metrics

metrics-json

3.1.5

io.dropwizard.metrics

metrics-jvm

3.1.5

io.dropwizard.metrics

metrics-log4j

3.1.5

io.dropwizard.metrics

metrics-servlets

3.1.5

io.netty

netty

3.9.9.Final

io.netty

netty-all

4.1.17.Final

io.prometheus

simpleclient

0.0.16

io.prometheus

simpleclient_common

0.0.16

io.prometheus

simpleclient_dropwizard

0.0.16

io.prometheus

simpleclient_servlet

0.0.16

io.prometheus.jmx

collector

0.7

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.validation

validation-api

1.1.0.Final

javax.ws.rs

javax.ws.rs-api

2.0.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.11

joda-time

joda-time

2.9.3

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

net.hydromatic

eigenbase-properties

1.1.5

net.iharder

base64

2.3.8

net.java.dev.jets3t

jets3t

0.9.4

net.razorvine

pyrolite

4.13

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt

oncrpc

1.0.7

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.4

org.antlr

antlr4-runtime

4.7

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

0.8.0

org.apache.arrow

arrow-memory

0.8.0

org.apache.arrow

arrow-vector

0.8.0

org.apache.avro

avro

1.7.7

org.apache.avro

avro-ipc

1.7.7

org.apache.avro

avro-ipc-tests

1.7.7

org.apache.avro

avro-mapred-hadoop2

1.7.7

org.apache.calcite

calcite-avatica

1.2.0-incubating

org.apache.calcite

calcite-core

1.2.0-incubating

org.apache.calcite

calcite-linq4j

1.2.0-incubating

org.apache.commons

commons-compress

1.4.1

org.apache.commons

commons-crypto

1.0.0

org.apache.commons

commons-lang3

3.5

org.apache.commons

commons-math3

3.4.1

org.apache.curator

curator-client

2.7.1

org.apache.curator

curator-framework

2.7.1

org.apache.curator

curator-recipes

2.7.1

org.apache.derby

derby

10.12.1.1

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.3

org.apache.hadoop

hadoop-auth

2.7.3

org.apache.hadoop

hadoop-client

2.7.3

org.apache.hadoop

hadoop-common

2.7.3

org.apache.hadoop

hadoop-hdfs

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.3

org.apache.hadoop

hadoop-yarn-api

2.7.3

org.apache.hadoop

hadoop-yarn-client

2.7.3

org.apache.hadoop

hadoop-yarn-common

2.7.3

org.apache.hadoop

hadoop-yarn-server-common

2.7.3

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.4

org.apache.httpcomponents

httpcore

4.4.8

org.apache.ivy

ivy

2.4.0

org.apache.orc

orc-core-nohive

1.4.1

org.apache.orc

orc-mapreduce-nohive

1.4.1

org.apache.parquet

parquet-column

1.8.2-databricks1

org.apache.parquet

parquet-common

1.8.2-databricks1

org.apache.parquet

parquet-encoding

1.8.2-databricks1

org.apache.parquet

parquet-format

2.3.1

org.apache.parquet

parquet-hadoop

1.8.2-databricks1

org.apache.parquet

parquet-jackson

1.8.2-databricks1

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.9.3

org.apache.xbean

xbean-asm5-shaded

4.4

org.apache.zookeeper

zookeeper

3.4.6

org.bouncycastle

bcprov-jdk15on

1.58

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.8

org.codehaus.janino

janino

3.0.8

org.datanucleus

datanucleus-api-jdo

3.2.6

org.datanucleus

datanucleus-core

3.2.10

org.datanucleus

datanucleus-rdbms

3.2.9

org.eclipse.jetty

jetty-client

9.3.20.v20170531

org.eclipse.jetty

jetty-continuation

9.3.20.v20170531

org.eclipse.jetty

jetty-http

9.3.20.v20170531

org.eclipse.jetty

jetty-io

9.3.20.v20170531

org.eclipse.jetty

jetty-jndi

9.3.20.v20170531

org.eclipse.jetty

jetty-plus

9.3.20.v20170531

org.eclipse.jetty

jetty-proxy

9.3.20.v20170531

org.eclipse.jetty

jetty-security

9.3.20.v20170531

org.eclipse.jetty

jetty-server

9.3.20.v20170531

org.eclipse.jetty

jetty-servlet

9.3.20.v20170531

org.eclipse.jetty

jetty-servlets

9.3.20.v20170531

org.eclipse.jetty

jetty-util

9.3.20.v20170531

org.eclipse.jetty

jetty-webapp

9.3.20.v20170531

org.eclipse.jetty

jetty-xml

9.3.20.v20170531

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.4.0-b34

org.glassfish.hk2

hk2-locator

2.4.0-b34

org.glassfish.hk2

hk2-utils

2.4.0-b34

org.glassfish.hk2

osgi-resource-locator

1.0.1

org.glassfish.hk2.external

aopalliance-repackaged

2.4.0-b34

org.glassfish.hk2.external

javax.inject

2.4.0-b34

org.glassfish.jersey.bundles.repackaged

jersey-guava

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet-core

2.22.2

org.glassfish.jersey.core

jersey-client

2.22.2

org.glassfish.jersey.core

jersey-common

2.22.2

org.glassfish.jersey.core

jersey-server

2.22.2

org.glassfish.jersey.media

jersey-media-jaxb

2.22.2

org.hibernate

hibernate-validator

5.1.1.Final

org.iq80.snappy

snappy

0.2

org.javassist

javassist

3.18.1-GA

org.jboss.logging

jboss-logging

3.1.3.GA

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.11

3.2.11

org.json4s

json4s-core_2.11

3.2.11

org.json4s

json4s-jackson_2.11

3.2.11

org.lz4

lz4-java

1.4.0

org.mariadb.jdbc

mariadb-java-client

2.1.2

org.mockito

mockito-all

1.9.5

org.objenesis

objenesis

2.1

org.postgresql

postgresql

42.1.4

org.roaringbitmap

RoaringBitmap

0.5.11

org.rocksdb

rocksdbjni

5.2.1

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.11

2.11.8

org.scala-lang

scala-library_2.11

2.11.8

org.scala-lang

scala-reflect_2.11

2.11.8

org.scala-lang

scalap_2.11

2.11.8

org.scala-lang.modules

scala-parser-combinators_2.11

1.0.2

org.scala-lang.modules

scala-xml_2.11

1.0.5

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.11

1.12.5

org.scalanlp

breeze-macros_2.11

0.13.2

org.scalanlp

breeze_2.11

0.13.2

org.scalatest

scalatest_2.11

2.2.6

org.slf4j

jcl-over-slf4j

1.7.16

org.slf4j

jul-to-slf4j

1.7.16

org.slf4j

slf4j-api

1.7.16

org.slf4j

slf4j-log4j12

1.7.16

org.spark-project.hive

hive-beeline

1.2.1.spark2

org.spark-project.hive

hive-cli

1.2.1.spark2

org.spark-project.hive

hive-exec

1.2.1.spark2

org.spark-project.hive

hive-jdbc

1.2.1.spark2

org.spark-project.hive

hive-metastore

1.2.1.spark2

org.spark-project.spark

unused

1.0.0

org.spire-math

spire-macros_2.11

0.13.0

org.spire-math

spire_2.11

0.13.0

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.tukaani

xz

1.0

org.typelevel

machinist_2.11

0.6.1

org.typelevel

macro-compat_2.11

1.1.1

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.2.6

org.yaml

snakeyaml

1.16

oro

oro

2.0.8

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1

xmlenc

xmlenc

0.52