2.1.0-db2cluster image

Databricks released this image in late January, 2017.

Important

This release has been deprecated. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks runtime support lifecycle.

The following release notes provide information about the Spark 2.1.0-db2 cluster image powered by Apache Spark.

Changes and Improvements

  • The Avro and Amazon Redshift data sources are now bundled in the cluster image. Please see the linked documentation on these data sources for more details.

  • Added Kafka 0.8 support for Structured Streaming. You can use format("kafka08") to read data from a Kafka 0.8+ cluster.

  • Added minPartitions option to Kafka 0.10 Streaming DataSource to configure parallelism when reading from Kafka.

  • Fixed a bug in the Amazon Redshift data source where the combination of filter pushdown and zero column selection (e.g. count(*) queries) caused runtime errors.

  • Changed the default file system for the s3a:// URL scheme to S3AFileSystem instead of DBFS.

Apache Spark

2.1.0-db2 cluster image includes the Apache Spark 2.1.0 release. You can consult JIRA for the detailed changes. 2.1.0-db2 cluster image also includes the following extra bug fixes and improvements:

  • [SPARK-4105][BACKPORT] retry the fetch or stage if shuffle block is corrupt

  • [SPARK-18917] Remove schema check in appending data

  • [SPARK-19314][SS][CATALYST] Do not allow sort before aggregation in Structured Streaming plan

  • [SPARK-19295][SQL] IsolatedClientLoader’s downloadVersion should log the location of downloaded metastore client jars

  • [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor the error checking when append data to an existing table

  • [SPARK-18475] Be able to increase parallelism in StructuredStreaming Kafka source

  • [SPARK-19168][STRUCTURED STREAMING] StateStore should be aborted upon error

  • [SPARK-19113][SS][TESTS] Ignore StreamingQueryException thrown from awaitInitialization to avoid breaking tests

  • [SPARK-19231][SPARKR] add error handling for download and untar for Spark release

  • [SPARK-19066][SPARKR][BACKPORT-2.1] LDA doesn’t set optimizer correctly

  • [SPARK-17755][CORE] Use workerRef to send RegisterWorkerResponse to avoid the race condition

  • [SPARK-19129][SQL] SessionCatalog: Disallow empty part col values in partition spec

  • [SPARK-19065][SQL] Don’t inherit expression id in dropDuplicates

  • [SPARK-19019][PYTHON] Fix hijacked collections.namedtuple and port cloudpickle changes for PySpark to work with Python 3.6.0

  • [SPARK-18905][STREAMING] Fix the issue of removing a failed jobset from JobScheduler.jobSets

  • [SPARK-19232][SPARKR] Update Spark distribution download cache location on Windows

  • [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parquet

  • [SPARK-19092][SQL][BACKPORT-2.1] Save() API of DataFrameWriter should not scan all the saved files #16481

  • [SPARK-19120] Refresh Metadata Cache After Loading Hive Tables

  • [SPARK-19180][SQL] the offset of short should be 2 in OffHeapColumn

  • [SPARK-18335][SPARKR] createDataFrame to support numPartitions parameter

  • [SPARK-19178][SQL] convert string of large numbers to int should return null

  • [SPARK-18687][PYSPARK][SQL] Backward compatibility - creating a Dataframe on a new SQLContext object fails with a Derby error

  • [SPARK-17237][SQL] Remove backticks in a pivot result schema

  • [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped

  • [SPARK-18969][SQL] Support grouping by nondeterministic expressions

  • [SPARK-18857][SQL] Don’t use Iterator.duplicate for incrementalCollect in Thrift Server

  • [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package.

  • [SPARK-19130][SPARKR] Support setting literal value as column implicitly

  • [SPARK-19133][SPARKR][ML][BACKPORT-2.1] fix glm for Gamma, clarify glm family supported

  • [SPARK-19140][SS] Allow update mode for non-aggregation streaming queries

  • [SPARK-18997][CORE] Recommended upgrade libthrift to 0.9.3

  • [SPARK-19113][SS][TESTS] Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization

  • [SPARK-19137][SQL] Fix withSQLConf to reset OptionalConfigEntry correctly

  • [SPARK-16845][SQL] GeneratedClass$SpecificOrdering grows beyond 64 KB

  • [SPARK-18952][BACKPORT] Regex strings not properly escaped in codegen for aggregations

  • [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to get SparkUI URL

  • [SPARK-19126][DOCS] Update Join Documentation Across Languages

  • [SPARK-19127][DOCS] Update Rank Function Documentation

  • [SPARK-18941][SQL][DOC] Add a new behavior document on CREATE/DROP TABLE with LOCATION

  • [SPARK-19106][DOCS] Styling for the configuration docs is broken

  • [SPARK-19110][ML][MLLIB] DistributedLDAModel returns different logPrior for original and loaded model

  • [SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options

  • [SPARK-19083] sbin/start-history-server.sh script use of $@ without quotes

  • [SPARK-19033][CORE] Add admin acls for history server

  • [SPARK-18877][SQL][BACKPORT-2.1] CSVInferSchema.inferField on DecimalType should find a common type with typeSoFar

  • [SPARK-19048][SQL] Delete Partition Location when Dropping Managed Partitioned Tables in InMemoryCatalog

  • [SPARK-19028][SQL] Fixed non-thread-safe functions used in SessionCatalog

  • [SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide

  • [SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.

  • [SPARK-19050][SS][TESTS] Fix EventTimeWatermarkSuite ‘delay in months and years handled correctly’

  • [SPARK-19016][SQL][DOC] Document scalable partition handling

  • [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD

  • [SPARK-18775][SQL] Limit the max number of records written per file

  • [SPARK-17910][SQL] Allow users to update the comment of a column

  • [SPARK-18669][SS][DOCS] Update Apache docs for Structured Streaming regarding watermarking and status

  • [SPARK-18993][BUILD] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags

  • [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI

  • [SPARK-18991][CORE] Change ContextCleaner.referenceBuffer to use ConcurrentHashMap to make it faster

  • [SPARK-18972][CORE] Fix the netty thread names for RPC

  • [SPARK-18985][SS] Add missing @InterfaceStability.Evolving for Structured Streaming APIs

  • [SPARK-17807][CORE] split test-tags into test-JAR

  • [SPARK-18973][SQL] Remove SortPartitions and RedistributeData

  • [SPARK-18908][SS] Creating StreamingQueryException should check if logicalPlan is created

  • [SPARK-18528][SQL] Fix a bug to initialise an iterator of aggregation buffer

  • [SPARK-18234][SS] Made update mode public

  • [SPARK-18588][SS][KAFKA] Create a new KafkaConsumer when error happens to fix the flaky test

  • [SPARK-18949][SQL][BACKPORT-2.1] Add recoverPartitions API to Catalog

  • [SPARK-18954][TESTS] Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window

  • [SPARK-18031][TESTS] Fix flaky test ExecutorAllocationManagerSuite.basic functionality

  • [SPARK-18894][SS] Fix event time watermark delay threshold specified in months or years

  • [SPARK-18947][SQL] SQLContext.tableNames should not call Catalog.listTables

  • [SPARK-18900][FLAKY-TEST] StateStoreSuite.maintenance

  • [SPARK-18927][SS] MemorySink for StructuredStreaming can’t recover from checkpoint if location is provided in SessionConf

  • [SPARK-18281][SQL] [PYSPARK] Remove timeout for reading data through socket for local iterator

  • [SPARK-18761][CORE] Introduce “task reaper” to oversee task killing in executors

  • [SPARK-18928] Check TaskContext.isInterrupted() in FileScanRDD, JDBCRDD & UnsafeSorter

  • [SPARK-18921][SQL] check database existence with Hive.databaseExists instead of getDatabase

  • [SPARK-18700][SQL] Add StripedLock for each table’s relation in cache

  • [SPARK-18703][SPARK-18675][SQL][BACKPORT-2.1] CTAS for hive serde table should work for all hive versions AND Drop Staging Directories and Data Files

  • [SPARK-18827][CORE] Fix cannot read broadcast on disk

  • [SPARK-18918][DOC] Missing </td> in Configuration page

  • [SPARK-18849][ML][SPARKR][DOC] vignettes final check reorg

  • [SPARK-18186][BRANCH-2.1] Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support

  • [SPARK-18904][SS][TESTS] Merge two FileStreamSourceSuite files

  • [SPARK-18897][SPARKR] Fix SparkR SQL Test to drop test table

  • [SPARK-18108][SQL] Fix a schema inconsistent bug that makes a parquet reader fail to read data

  • [SPARK-18850][SS] Make StreamExecution and progress classes serializable

  • [SPARK-18892][SQL] Alias percentile_approx approx_percentile

Known Issues

  • Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

System Environment

  • Operating System: Ubuntu 16.04.1 LTS

  • Java: 1.8.0_111

  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)

  • Python: 2.7.12 (or 3.5.2 if using Python 3)

  • R: R version 3.2.3 (2015-12-10)

Pre-installed Python Libraries

Library

Version

Library

Version

Library

Version

ansi2html

1.1.1

argparse

1.2.1

boto

2.42.0

boto3

1.4.1

botocore

1.4.70

brewer2mpl

1.4.1

certifi

2016.2.28

cffi

1.7.0

chardet

2.3.0

colorama

0.3.7

configobj

5.0.6

cryptography

1.5

cycler

0.10.0

Cython

0.24.1

decorator

4.0.10

docutils

0.13.1

enum34

1.1.6

et-xmlfile

1.0.1

freetype-py

1.0.2

funcsigs

1.0.2

fusepy

2.0.4

futures

3.0.5

ggplot

0.6.8

html5lib

0.999

idna

2.1

ipaddress

1.0.16

ipython

2.2.0

ipython-genutils

0.1.0

jdcal

1.2

Jinja2

2.8

jmespath

0.9.0

llvmlite

0.13.0

lxml

3.6.4

MarkupSafe

0.23

matplotlib

1.5.3

mpld3

0.2

msgpack-python

0.4.7

ndg-httpsclient

0.3.3

numba

0.28.1

numpy

1.11.1

openpyxl

2.3.2

pandas

0.18.1

pathlib2

2.1.0

patsy

0.4.1

pexpect

4.0.1

pickleshare

0.7.4

Pillow

3.3.1

pip

9.0.1

pkg_resources

0.0.0

ply

3.9

prompt-toolkit

1.0.7

psycopg2

2.6.2

ptyprocess

0.5.1

py4j

0.10.3

pyasn1

0.1.9

pycparser

2.14

Pygments

2.1.3

PyGObject

3.20.0

pyOpenSSL

16.0.0

pyparsing

2.1.4

pypng

0.0.18

Python

2.7.12

python-dateutil

2.5.3

python-geohash

0.8.5

pytz

2016.6.1

requests

2.11.1

s3transfer

0.1.9

scikit-learn

0.17.1

scipy

0.18.1

scour

0.32

seaborn

0.7.1

setuptools

32.3.1

simplejson

3.8.2

simples3

1.0

singledispatch

3.4.0.3

six

1.10.0

statsmodels

0.6.1

traitlets

4.3.0

urllib3

1.19.1

virtualenv

15.0.1

wcwidth

0.1.7

wheel

0.30.0a0

wsgiref

0.1.2

Pre-installed R Libraries

Library

Version

Library

Version

Library

Version

abind

1.4-3

assertthat

0.1

base

3.2.3

BH

1.60.0-2

bitops

1.0-6

boot

1.3-17

brew

1.0-6

car

2.1-3

caret

6.0-71

chron

2.3-47

class

7.3-14

cluster

2.0.5

codetools

0.2-14

colorspace

1.2-4

compiler

3.2.3

crayon

1.3.1

curl

2.2

data.table

1.9.6

datasets

3.2.3

DBI

0.5-1

devtools

1.12.0

dichromat

2.0-0

digest

0.6.9

doMC

1.3.4

dplyr

0.5.0

foreach

1.4.3

foreign

0.8-66

gbm

2.1.1

ggplot2

2.1.0

git2r

0.15.0

glmnet

2.0-5

graphics

3.2.3

grDevices

3.2.3

grid

3.2.3

gsubfn

0.6-6

gtable

0.1.2

h2o

3.10.0.8

httr

1.2.1

hwriter

1.3.2

hwriterPlus

1.0-3

iterators

1.0.8

jsonlite

1.1

KernSmooth

2.23-15

labeling

0.3

lattice

0.20-34

lazyeval

0.2.0

littler

0.3.0

lme4

1.1-12

lubridate

1.6.0

magrittr

1.5

mapproj

1.2-4

maps

3.0.2

MASS

7.3-45

Matrix

1.2-7.1

MatrixModels

0.4-1

memoise

1.0.0

methods

3.2.3

mgcv

1.8-11

mime

0.5

minqa

1.2.4

multicore

0.2

munsell

0.4.2

mvtnorm

1.0-5

nlme

3.1-124

nloptr

1.0.4

nnet

7.3-12

openssl

0.9.4

parallel

3.2.3

pbkrtest

0.4-6

pkgKitten

0.1.3

plyr

1.8.4

praise

1.0.0

pROC

1.8

proto

0.3-10

quantreg

5.29

R.methodsS3

1.7.1

R.oo

1.20.0

R.utils

2.4.0

R6

2.2.0

randomForest

4.6-12

RColorBrewer

1.1-2

Rcpp

0.12.7

RcppEigen

0.3.2.9.0

RCurl

1.95-4.8

reshape2

1.4.2

RODBC

1.3-12

roxygen2

5.0.1

rpart

4.1-10

Rserve

1.7-3

RSQLite

1.0.0

rstudioapi

0.6

scales

0.3.0

sp

1.0-15

SparkR

2.1.0

SparseM

1.72

spatial

7.3-11

splines

3.2.3

sqldf

0.4-10

statmod

1.4.26

stats

3.2.3

stats4

3.2.3

stringi

1.0-1

stringr

1.0.0

survival

2.38-3

tcltk

3.2.3

TeachingDemos

2.10

testthat

1.0.2

tibble

1.2

tools

3.2.3

utils

3.2.3

whisker

0.3-2

withr

1.0.2

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

aws-java-sdk

1.9.40

com.amazonaws

aws-java-sdk-autoscaling

1.9.40

com.amazonaws

aws-java-sdk-cloudformation

1.9.40

com.amazonaws

aws-java-sdk-cloudfront

1.9.40

com.amazonaws

aws-java-sdk-cloudhsm

1.9.40

com.amazonaws

aws-java-sdk-cloudsearch

1.9.40

com.amazonaws

aws-java-sdk-cloudtrail

1.9.40

com.amazonaws

aws-java-sdk-cloudwatch

1.9.40

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.9.40

com.amazonaws

aws-java-sdk-codedeploy

1.9.40

com.amazonaws

aws-java-sdk-cognitoidentity

1.9.40

com.amazonaws

aws-java-sdk-cognitosync

1.9.40

com.amazonaws

aws-java-sdk-config

1.9.40

com.amazonaws

aws-java-sdk-core

1.9.40

com.amazonaws

aws-java-sdk-datapipeline

1.9.40

com.amazonaws

aws-java-sdk-directconnect

1.9.40

com.amazonaws

aws-java-sdk-directory

1.9.40

com.amazonaws

aws-java-sdk-dynamodb

1.9.40

com.amazonaws

aws-java-sdk-ec2

1.9.40

com.amazonaws

aws-java-sdk-ecs

1.9.40

com.amazonaws

aws-java-sdk-efs

1.9.40

com.amazonaws

aws-java-sdk-elasticache

1.9.40

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.9.40

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.9.40

com.amazonaws

aws-java-sdk-elastictranscoder

1.9.40

com.amazonaws

aws-java-sdk-emr

1.9.40

com.amazonaws

aws-java-sdk-glacier

1.9.40

com.amazonaws

aws-java-sdk-iam

1.9.40

com.amazonaws

aws-java-sdk-importexport

1.9.40

com.amazonaws

aws-java-sdk-kinesis

1.9.40

com.amazonaws

aws-java-sdk-kms

1.9.40

com.amazonaws

aws-java-sdk-lambda

1.9.40

com.amazonaws

aws-java-sdk-logs

1.9.40

com.amazonaws

aws-java-sdk-machinelearning

1.9.40

com.amazonaws

aws-java-sdk-opsworks

1.9.40

com.amazonaws

aws-java-sdk-rds

1.9.40

com.amazonaws

aws-java-sdk-redshift

1.9.40

com.amazonaws

aws-java-sdk-route53

1.9.40

com.amazonaws

aws-java-sdk-s3

1.9.40

com.amazonaws

aws-java-sdk-ses

1.9.40

com.amazonaws

aws-java-sdk-simpledb

1.9.40

com.amazonaws

aws-java-sdk-simpleworkflow

1.9.40

com.amazonaws

aws-java-sdk-sns

1.9.40

com.amazonaws

aws-java-sdk-sqs

1.9.40

com.amazonaws

aws-java-sdk-ssm

1.9.40

com.amazonaws

aws-java-sdk-storagegateway

1.9.40

com.amazonaws

aws-java-sdk-sts

1.9.40

com.amazonaws

aws-java-sdk-support

1.9.40

com.amazonaws

aws-java-sdk-swf-libraries

1.9.40

com.amazonaws

aws-java-sdk-workspaces

1.9.40

com.chuusai

shapeless_2.10.4

2.0.0

com.clearspring.analytics

stream

2.7.0

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.10

0.4.15-9

com.databricks.scalapb

scalapb-runtime_2.10

0.4.15-9

com.esotericsoftware

kryo-shaded

3.0.3

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.0.0

com.fasterxml.jackson.core

jackson-annotations

2.4.5

com.fasterxml.jackson.core

jackson-core

2.4.5

com.fasterxml.jackson.core

jackson-databind

2.4.5

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.4.5

com.fasterxml.jackson.module

jackson-module-paranamer

2.4.5

com.fasterxml.jackson.module

jackson-module-scala_2.10

2.4.5

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1

com.github.rwl

jtransforms

2.4.0

com.google.code.findbugs

jsr305

2.0.1

com.google.code.gson

gson

2.2.4

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.googlecode.javaewah

JavaEWAH

0.3.2

com.h2database

h2

1.3.174

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.mchange

c3p0

0.9.5.1

com.mchange

mchange-commons-java

0.2.10

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.thoughtworks.paranamer

paranamer

2.6

com.trueaccord.lenses

lenses_2.10

0.3

com.twitter

chill-java

0.8.0

com.twitter

chill_2.10

0.8.0

com.twitter

parquet-hadoop-bundle

1.6.0

com.twitter

util-app_2.10

6.23.0

com.twitter

util-core_2.10

6.23.0

com.twitter

util-jvm_2.10

6.23.0

com.typesafe

config

1.2.1

com.typesafe

scalalogging-slf4j_2.10

1.1.0

com.univocity

univocity-parsers

2.2.1

com.zaxxer

HikariCP

2.4.1

commons-beanutils

commons-beanutils

1.7.0

commons-beanutils

commons-beanutils-core

1.8.0

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

2.2

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.7

io.dropwizard.metrics

metrics-core

3.1.2

io.dropwizard.metrics

metrics-ganglia

3.1.2

io.dropwizard.metrics

metrics-graphite

3.1.2

io.dropwizard.metrics

metrics-healthchecks

3.1.2

io.dropwizard.metrics

metrics-jetty9

3.1.2

io.dropwizard.metrics

metrics-json

3.1.2

io.dropwizard.metrics

metrics-jvm

3.1.2

io.dropwizard.metrics

metrics-log4j

3.1.2

io.dropwizard.metrics

metrics-servlets

3.1.2

io.netty

netty

3.8.0.Final

io.netty

netty-all

4.0.42.Final

io.prometheus

simpleclient

0.0.16

io.prometheus

simpleclient_common

0.0.16

io.prometheus

simpleclient_dropwizard

0.0.16

io.prometheus

simpleclient_servlet

0.0.16

io.prometheus.jmx

collector

0.7

javax.activation

activation

1.1

javax.annotation

javax.annotation-api

1.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.validation

validation-api

1.1.0.Final

javax.ws.rs

javax.ws.rs-api

2.0.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.11

joda-time

joda-time

2.9.3

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

mysql

mysql-connector-java

5.1.27

net.hydromatic

eigenbase-properties

1.1.5

net.java.dev.jets3t

jets3t

0.7.1

net.jpountz.lz4

lz4

1.3.0

net.razorvine

pyrolite

4.13

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.py4j

py4j

0.10.4

net.sf.supercsv

super-csv

2.2.0

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt

oncrpc

1.0.7

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.4

org.antlr

antlr4-runtime

4.5.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.avro

avro

1.7.7

org.apache.avro

avro-ipc

1.7.7

org.apache.avro

avro-mapred

1.7.7

org.apache.calcite

calcite-avatica

1.2.0-incubating

org.apache.calcite

calcite-core

1.2.0-incubating

org.apache.calcite

calcite-linq4j

1.2.0-incubating

org.apache.commons

commons-compress

1.4.1

org.apache.commons

commons-crypto

1.0.0

org.apache.commons

commons-lang3

3.5

org.apache.commons

commons-math3

3.4.1

org.apache.curator

curator-client

2.6.0

org.apache.curator

curator-framework

2.6.0

org.apache.curator

curator-recipes

2.6.0

org.apache.derby

derby

10.10.2.0

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.3

org.apache.hadoop

hadoop-auth

2.7.3

org.apache.hadoop

hadoop-client

2.7.3

org.apache.hadoop

hadoop-common

2.7.3

org.apache.hadoop

hadoop-hdfs

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.3

org.apache.hadoop

hadoop-yarn-api

2.7.3

org.apache.hadoop

hadoop-yarn-client

2.7.3

org.apache.hadoop

hadoop-yarn-common

2.7.3

org.apache.hadoop

hadoop-yarn-server-common

2.7.3

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.2

org.apache.httpcomponents

httpcore

4.4.4

org.apache.ivy

ivy

2.4.0

org.apache.parquet

parquet-column

1.8.1

org.apache.parquet

parquet-common

1.8.1

org.apache.parquet

parquet-encoding

1.8.1

org.apache.parquet

parquet-format

2.3.0-incubating

org.apache.parquet

parquet-hadoop

1.8.1

org.apache.parquet

parquet-jackson

1.8.1

org.apache.thrift

libfb303

0.9.2

org.apache.thrift

libthrift

0.9.2

org.apache.xbean

xbean-asm5-shaded

4.4

org.apache.zookeeper

zookeeper

3.4.6

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.0

org.codehaus.janino

janino

3.0.0

org.datanucleus

datanucleus-api-jdo

3.2.6

org.datanucleus

datanucleus-core

3.2.10

org.datanucleus

datanucleus-rdbms

3.2.9

org.eclipse.jetty

jetty-client

9.3.3.v20150827

org.eclipse.jetty

jetty-continuation

9.3.3.v20150827

org.eclipse.jetty

jetty-http

9.3.3.v20150827

org.eclipse.jetty

jetty-io

9.3.3.v20150827

org.eclipse.jetty

jetty-jndi

9.3.3.v20150827

org.eclipse.jetty

jetty-plus

9.3.3.v20150827

org.eclipse.jetty

jetty-proxy

9.3.3.v20150827

org.eclipse.jetty

jetty-security

9.3.3.v20150827

org.eclipse.jetty

jetty-server

9.3.3.v20150827

org.eclipse.jetty

jetty-servlet

9.3.3.v20150827

org.eclipse.jetty

jetty-servlets

9.3.3.v20150827

org.eclipse.jetty

jetty-util

9.3.3.v20150827

org.eclipse.jetty

jetty-webapp

9.3.3.v20150827

org.eclipse.jetty

jetty-xml

9.3.3.v20150827

org.fusesource.jansi

jansi

1.4

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.4.0-b34

org.glassfish.hk2

hk2-locator

2.4.0-b34

org.glassfish.hk2

hk2-utils

2.4.0-b34

org.glassfish.hk2

osgi-resource-locator

1.0.1

org.glassfish.hk2.external

aopalliance-repackaged

2.4.0-b34

org.glassfish.hk2.external

javax.inject

2.4.0-b34

org.glassfish.jersey.bundles.repackaged

jersey-guava

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet-core

2.22.2

org.glassfish.jersey.core

jersey-client

2.22.2

org.glassfish.jersey.core

jersey-common

2.22.2

org.glassfish.jersey.core

jersey-server

2.22.2

org.glassfish.jersey.media

jersey-media-jaxb

2.22.2

org.hibernate

hibernate-validator

5.1.1.Final

org.iq80.snappy

snappy

0.2

org.javassist

javassist

3.18.1-GA

org.jboss.logging

jboss-logging

3.1.3.GA

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.jpmml

pmml-model

1.2.15

org.jpmml

pmml-schema

1.2.15

org.json4s

json4s-ast_2.10

3.2.11

org.json4s

json4s-core_2.10

3.2.11

org.json4s

json4s-jackson_2.10

3.2.11

org.mockito

mockito-all

1.9.5

org.objenesis

objenesis

2.1

org.postgresql

postgresql

9.4-1204-jdbc41

org.roaringbitmap

RoaringBitmap

0.5.11

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

jline

2.10.6

org.scala-lang

scala-compiler_2.10

2.10.6

org.scala-lang

scala-library_2.10

2.10.6

org.scala-lang

scala-reflect_2.10

2.10.6

org.scala-lang

scalap_2.10

2.10.6

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.10

1.12.5

org.scalamacros

quasiquotes_2.10

2.0.0

org.scalanlp

breeze-macros_2.10

0.12

org.scalanlp

breeze_2.10

0.12

org.scalatest

scalatest_2.10

2.2.6

org.slf4j

jcl-over-slf4j

1.7.16

org.slf4j

jul-to-slf4j

1.7.16

org.slf4j

slf4j-api

1.7.16

org.slf4j

slf4j-log4j12

1.7.16

org.spark-project.hive

hive-beeline

1.2.1.spark2

org.spark-project.hive

hive-cli

1.2.1.spark2

org.spark-project.hive

hive-exec

1.2.1.spark2

org.spark-project.hive

hive-jdbc

1.2.1.spark2

org.spark-project.hive

hive-metastore

1.2.1.spark2

org.spark-project.spark

unused

1.0.0

org.spire-math

spire-macros_2.10

0.7.4

org.spire-math

spire_2.10

0.7.4

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.tukaani

xz

1.0

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.2.6

org.yaml

snakeyaml

1.16

oro

oro

2.0.8

stax

stax-api

1.0.1

xerces

xercesImpl

2.9.1

xml-apis

xml-apis

1.3.04

xmlenc

xmlenc

0.52

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

aws-java-sdk

1.9.40

com.amazonaws

aws-java-sdk-autoscaling

1.9.40

com.amazonaws

aws-java-sdk-cloudformation

1.9.40

com.amazonaws

aws-java-sdk-cloudfront

1.9.40

com.amazonaws

aws-java-sdk-cloudhsm

1.9.40

com.amazonaws

aws-java-sdk-cloudsearch

1.9.40

com.amazonaws

aws-java-sdk-cloudtrail

1.9.40

com.amazonaws

aws-java-sdk-cloudwatch

1.9.40

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.9.40

com.amazonaws

aws-java-sdk-codedeploy

1.9.40

com.amazonaws

aws-java-sdk-cognitoidentity

1.9.40

com.amazonaws

aws-java-sdk-cognitosync

1.9.40

com.amazonaws

aws-java-sdk-config

1.9.40

com.amazonaws

aws-java-sdk-core

1.9.40

com.amazonaws

aws-java-sdk-datapipeline

1.9.40

com.amazonaws

aws-java-sdk-directconnect

1.9.40

com.amazonaws

aws-java-sdk-directory

1.9.40

com.amazonaws

aws-java-sdk-dynamodb

1.9.40

com.amazonaws

aws-java-sdk-ec2

1.9.40

com.amazonaws

aws-java-sdk-ecs

1.9.40

com.amazonaws

aws-java-sdk-efs

1.9.40

com.amazonaws

aws-java-sdk-elasticache

1.9.40

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.9.40

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.9.40

com.amazonaws

aws-java-sdk-elastictranscoder

1.9.40

com.amazonaws

aws-java-sdk-emr

1.9.40

com.amazonaws

aws-java-sdk-glacier

1.9.40

com.amazonaws

aws-java-sdk-iam

1.9.40

com.amazonaws

aws-java-sdk-importexport

1.9.40

com.amazonaws

aws-java-sdk-kinesis

1.9.40

com.amazonaws

aws-java-sdk-kms

1.9.40

com.amazonaws

aws-java-sdk-lambda

1.9.40

com.amazonaws

aws-java-sdk-logs

1.9.40

com.amazonaws

aws-java-sdk-machinelearning

1.9.40

com.amazonaws

aws-java-sdk-opsworks

1.9.40

com.amazonaws

aws-java-sdk-rds

1.9.40

com.amazonaws

aws-java-sdk-redshift

1.9.40

com.amazonaws

aws-java-sdk-route53

1.9.40

com.amazonaws

aws-java-sdk-s3

1.9.40

com.amazonaws

aws-java-sdk-ses

1.9.40

com.amazonaws

aws-java-sdk-simpledb

1.9.40

com.amazonaws

aws-java-sdk-simpleworkflow

1.9.40

com.amazonaws

aws-java-sdk-sns

1.9.40

com.amazonaws

aws-java-sdk-sqs

1.9.40

com.amazonaws

aws-java-sdk-ssm

1.9.40

com.amazonaws

aws-java-sdk-storagegateway

1.9.40

com.amazonaws

aws-java-sdk-sts

1.9.40

com.amazonaws

aws-java-sdk-support

1.9.40

com.amazonaws

aws-java-sdk-swf-libraries

1.9.40

com.amazonaws

aws-java-sdk-workspaces

1.9.40

com.chuusai

shapeless_2.11

2.0.0

com.clearspring.analytics

stream

2.7.0

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.11

0.4.15-9

com.databricks.scalapb

scalapb-runtime_2.11

0.4.15-9

com.esotericsoftware

kryo-shaded

3.0.3

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.0.0

com.fasterxml.jackson.core

jackson-annotations

2.4.5

com.fasterxml.jackson.core

jackson-core

2.4.5

com.fasterxml.jackson.core

jackson-databind

2.4.5

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.4.5

com.fasterxml.jackson.module

jackson-module-paranamer

2.4.5

com.fasterxml.jackson.module

jackson-module-scala_2.11

2.4.5

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1

com.github.rwl

jtransforms

2.4.0

com.google.code.findbugs

jsr305

2.0.1

com.google.code.gson

gson

2.2.4

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.googlecode.javaewah

JavaEWAH

0.3.2

com.h2database

h2

1.3.174

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.mchange

c3p0

0.9.5.1

com.mchange

mchange-commons-java

0.2.10

com.ning

compress-lzf

1.0.3

com.sun.mail

javax.mail

1.5.2

com.thoughtworks.paranamer

paranamer

2.6

com.trueaccord.lenses

lenses_2.11

0.3

com.twitter

chill-java

0.8.0

com.twitter

chill_2.11

0.8.0

com.twitter

parquet-hadoop-bundle

1.6.0

com.twitter

util-app_2.11

6.23.0

com.twitter

util-core_2.11

6.23.0

com.twitter

util-jvm_2.11

6.23.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging-api_2.11

2.1.2

com.typesafe.scala-logging

scala-logging-slf4j_2.11

2.1.2

com.univocity

univocity-parsers

2.2.1

com.zaxxer

HikariCP

2.4.1

commons-beanutils

commons-beanutils

1.7.0

commons-beanutils

commons-beanutils-core

1.8.0

commons-cli

commons-cli

1.2

commons-codec

commons-codec

1.10

commons-collections

commons-collections

3.2.2

commons-configuration

commons-configuration

1.6

commons-dbcp

commons-dbcp

1.4

commons-digester

commons-digester

1.8

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.4

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-net

commons-net

2.2

commons-pool

commons-pool

1.5.4

info.ganglia.gmetric4j

gmetric4j

1.0.7

io.dropwizard.metrics

metrics-core

3.1.2

io.dropwizard.metrics

metrics-ganglia

3.1.2

io.dropwizard.metrics

metrics-graphite

3.1.2

io.dropwizard.metrics

metrics-healthchecks

3.1.2

io.dropwizard.metrics

metrics-jetty9

3.1.2

io.dropwizard.metrics

metrics-json

3.1.2

io.dropwizard.metrics

metrics-jvm

3.1.2

io.dropwizard.metrics

metrics-log4j

3.1.2

io.dropwizard.metrics

metrics-servlets

3.1.2

io.netty

netty

3.8.0.Final

io.netty

netty-all

4.0.42.Final

io.prometheus

simpleclient

0.0.16

io.prometheus

simpleclient_common

0.0.16

io.prometheus

simpleclient_dropwizard

0.0.16

io.prometheus

simpleclient_servlet

0.0.16

io.prometheus.jmx

collector

0.7

javax.activation

activation

1.1

javax.annotation

javax.annotation-api

1.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.servlet

javax.servlet-api

3.1.0

javax.servlet.jsp

jsp-api

2.1

javax.transaction

jta

1.1

javax.validation

validation-api

1.1.0.Final

javax.ws.rs

javax.ws.rs-api

2.0.1

javax.xml.bind

jaxb-api

2.2.2

javax.xml.stream

stax-api

1.0-2

javolution

javolution

5.5.1

jline

jline

2.11

joda-time

joda-time

2.9.3

log4j

apache-log4j-extras

1.2.17

log4j

log4j

1.2.17

mysql

mysql-connector-java

5.1.27

net.hydromatic

eigenbase-properties

1.1.5

net.java.dev.jets3t

jets3t

0.7.1

net.jpountz.lz4

lz4

1.3.0

net.razorvine

pyrolite

4.13

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.py4j

py4j

0.10.4

net.sf.supercsv

super-csv

2.2.0

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt

oncrpc

1.0.7

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.4

org.antlr

antlr4-runtime

4.5.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.avro

avro

1.7.7

org.apache.avro

avro-ipc

1.7.7

org.apache.avro

avro-mapred

1.7.7

org.apache.calcite

calcite-avatica

1.2.0-incubating

org.apache.calcite

calcite-core

1.2.0-incubating

org.apache.calcite

calcite-linq4j

1.2.0-incubating

org.apache.commons

commons-compress

1.4.1

org.apache.commons

commons-crypto

1.0.0

org.apache.commons

commons-lang3

3.5

org.apache.commons

commons-math3

3.4.1

org.apache.curator

curator-client

2.6.0

org.apache.curator

curator-framework

2.6.0

org.apache.curator

curator-recipes

2.6.0

org.apache.derby

derby

10.10.2.0

org.apache.directory.api

api-asn1-api

1.0.0-M20

org.apache.directory.api

api-util

1.0.0-M20

org.apache.directory.server

apacheds-i18n

2.0.0-M15

org.apache.directory.server

apacheds-kerberos-codec

2.0.0-M15

org.apache.hadoop

hadoop-annotations

2.7.3

org.apache.hadoop

hadoop-auth

2.7.3

org.apache.hadoop

hadoop-client

2.7.3

org.apache.hadoop

hadoop-common

2.7.3

org.apache.hadoop

hadoop-hdfs

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-app

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-common

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-core

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-jobclient

2.7.3

org.apache.hadoop

hadoop-mapreduce-client-shuffle

2.7.3

org.apache.hadoop

hadoop-yarn-api

2.7.3

org.apache.hadoop

hadoop-yarn-client

2.7.3

org.apache.hadoop

hadoop-yarn-common

2.7.3

org.apache.hadoop

hadoop-yarn-server-common

2.7.3

org.apache.htrace

htrace-core

3.1.0-incubating

org.apache.httpcomponents

httpclient

4.5.2

org.apache.httpcomponents

httpcore

4.4.4

org.apache.ivy

ivy

2.4.0

org.apache.parquet

parquet-column

1.8.1

org.apache.parquet

parquet-common

1.8.1

org.apache.parquet

parquet-encoding

1.8.1

org.apache.parquet

parquet-format

2.3.0-incubating

org.apache.parquet

parquet-hadoop

1.8.1

org.apache.parquet

parquet-jackson

1.8.1

org.apache.thrift

libfb303

0.9.2

org.apache.thrift

libthrift

0.9.2

org.apache.xbean

xbean-asm5-shaded

4.4

org.apache.zookeeper

zookeeper

3.4.6

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-jaxrs

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.jackson

jackson-xc

1.9.13

org.codehaus.janino

commons-compiler

3.0.0

org.codehaus.janino

janino

3.0.0

org.datanucleus

datanucleus-api-jdo

3.2.6

org.datanucleus

datanucleus-core

3.2.10

org.datanucleus

datanucleus-rdbms

3.2.9

org.eclipse.jetty

jetty-client

9.3.3.v20150827

org.eclipse.jetty

jetty-continuation

9.3.3.v20150827

org.eclipse.jetty

jetty-http

9.3.3.v20150827

org.eclipse.jetty

jetty-io

9.3.3.v20150827

org.eclipse.jetty

jetty-jndi

9.3.3.v20150827

org.eclipse.jetty

jetty-plus

9.3.3.v20150827

org.eclipse.jetty

jetty-proxy

9.3.3.v20150827

org.eclipse.jetty

jetty-security

9.3.3.v20150827

org.eclipse.jetty

jetty-server

9.3.3.v20150827

org.eclipse.jetty

jetty-servlet

9.3.3.v20150827

org.eclipse.jetty

jetty-servlets

9.3.3.v20150827

org.eclipse.jetty

jetty-util

9.3.3.v20150827

org.eclipse.jetty

jetty-webapp

9.3.3.v20150827

org.eclipse.jetty

jetty-xml

9.3.3.v20150827

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.4.0-b34

org.glassfish.hk2

hk2-locator

2.4.0-b34

org.glassfish.hk2

hk2-utils

2.4.0-b34

org.glassfish.hk2

osgi-resource-locator

1.0.1

org.glassfish.hk2.external

aopalliance-repackaged

2.4.0-b34

org.glassfish.hk2.external

javax.inject

2.4.0-b34

org.glassfish.jersey.bundles.repackaged

jersey-guava

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet

2.22.2

org.glassfish.jersey.containers

jersey-container-servlet-core

2.22.2

org.glassfish.jersey.core

jersey-client

2.22.2

org.glassfish.jersey.core

jersey-common

2.22.2

org.glassfish.jersey.core

jersey-server

2.22.2

org.glassfish.jersey.media

jersey-media-jaxb

2.22.2

org.hibernate

hibernate-validator

5.1.1.Final

org.iq80.snappy

snappy

0.2

org.javassist

javassist

3.18.1-GA

org.jboss.logging

jboss-logging

3.1.3.GA

org.jdbi

jdbi

2.63.1

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.jpmml

pmml-model

1.2.15

org.jpmml

pmml-schema

1.2.15

org.json4s

json4s-ast_2.11

3.2.11

org.json4s

json4s-core_2.11

3.2.11

org.json4s

json4s-jackson_2.11

3.2.11

org.mockito

mockito-all

1.9.5

org.objenesis

objenesis

2.1

org.postgresql

postgresql

9.4-1204-jdbc41

org.roaringbitmap

RoaringBitmap

0.5.11

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.11

2.11.8

org.scala-lang

scala-library_2.11

2.11.8

org.scala-lang

scala-reflect_2.11

2.11.8

org.scala-lang

scalap_2.11

2.11.8

org.scala-lang.modules

scala-parser-combinators_2.11

1.0.2

org.scala-lang.modules

scala-xml_2.11

1.0.2

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.11

1.12.5

org.scalanlp

breeze-macros_2.11

0.12

org.scalanlp

breeze_2.11

0.12

org.scalatest

scalatest_2.11

2.2.6

org.slf4j

jcl-over-slf4j

1.7.16

org.slf4j

jul-to-slf4j

1.7.16

org.slf4j

slf4j-api

1.7.16

org.slf4j

slf4j-log4j12

1.7.16

org.spark-project.hive

hive-beeline

1.2.1.spark2

org.spark-project.hive

hive-cli

1.2.1.spark2

org.spark-project.hive

hive-exec

1.2.1.spark2

org.spark-project.hive

hive-jdbc

1.2.1.spark2

org.spark-project.hive

hive-metastore

1.2.1.spark2

org.spark-project.spark

unused

1.0.0

org.spire-math

spire-macros_2.11

0.7.4

org.spire-math

spire_2.11

0.7.4

org.springframework

spring-core

4.1.4.RELEASE

org.springframework

spring-test

4.1.4.RELEASE

org.tukaani

xz

1.0

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.2.6

org.yaml

snakeyaml

1.16

oro

oro

2.0.8

stax

stax-api

1.0.1

xerces

xercesImpl

2.9.1

xml-apis

xml-apis

1.3.04

xmlenc

xmlenc

0.52