2.1.1-db4 Cluster Image

Databricks released this image in late April, 2017.

Important

This release was deprecated on July 30, 2018. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks Runtime Versioning and Deprecation Policy.

The following release notes provide information about the 2.1.1-db4 cluster image powered by Apache Spark.

Changes and Improvements

  • Starting from this version, we will be rolling out an upgrade of the Databricks File System (DBFS) to our customers. This upgrade mainly consists of stability fixes to DBFS. The change is source code compatible, so no actions are required from our customers.
  • Added support of transactional writes to cloud storage for Spark jobs as part of the Databricks DBIO package.
  • Added support for Hive metastores created by Apache Hive 2.0.0 to 2.1.1. Please see External Hive Metastore for detailed instructions on how to connect Databricks clusters to an externally hosted Hive metastore.
  • Fixed an issue that Structured Streaming metadata files may be corrupted when a query is stopped.
  • Fixed the bug that caused R Notebooks loose connection with R process if they were left idle for a few hours.
  • Bug fixes and stability improvements to Spark.

Apache Spark

The 2.1.1-db4 cluster image includes Apache Spark 2.1.1 RC3. In addition to 2.1.0-db3 Cluster Image, the 2.1.1-db4 cluster image also includes the following extra bug fixes and improvements made to Spark:

  • [SPARK-20404][CORE] Using Option(name) instead of Some(name)
  • [SPARK-20451] Filter out nested mapType datatypes from sort order in randomSplit
  • [SPARK-20450][SQL] Unexpected first-query schema inference cost with 2.1.1
  • [SPARK-20409][SQL] fail early if aggregate function in GROUP BY
  • [SPARK-20359][SQL] Avoid unnecessary execution in EliminateOuterJoin optimization that can lead to NPE
  • [SPARK-17647][SQL][FOLLOWUP][MINOR] fix typo
  • [SPARK-20349][SQL][REVERT-BRANCH2.1] ListFunctions returns duplicate functions after using persistent functions
  • [SPARK-17647][SQL] Fix backslash escaping in ‘LIKE’ patterns.
  • [SPARK-20349][SQL] ListFunctions returns duplicate functions after using persistent functions
  • [SPARK-20335][SQL][BACKPORT-2.1] Children expressions of Hive UDF impacts the determinism of Hive UDF
  • [SPARK-19924][SQL][BACKPORT-2.1] Handle InvocationTargetException for all Hive Shim
  • [SPARK-20131][CORE] Don’t use this lock in StandaloneSchedulerBackend.stop
  • [SPARK-20304][SQL] AssertNotNull should not include path in string representation
  • [SPARK-20291][SQL] NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, DoubleType)
  • [SPARK-18555][MINOR][SQL] Fix the @since tag when backporting from 2.2 branch into 2.1 branch
  • [SPARK-20270][SQL] na.fill should not change the values in long or integer when the default value is in double
  • [SPARK-18555][SQL] DataFrameNaFunctions.fill miss up original values in long integers
  • [SPARK-20280][CORE] FileStatusCache Weigher integer overflow
  • [SPARK-20264][SQL] asm should be non-test dependency in sql/core
  • [SPARK-20260][MLLIB] String interpolation required for error message
  • [SPARK-20262][SQL] AssertNotNull should throw NullPointerException
  • [SPARK-20246][SQL] should not push predicate down through aggregate with non-deterministic expressions
  • [SPARK-18112][SPARK-19924][SQL][BACKPORT] Support reading data from Hive 2.1 metastore
  • [SPARK-20214][ML] Make sure converted csc matrix has sorted indices
  • [SPARK-20223][SQL] Fix typo in tpcds q77.sql
  • [SPARK-20042][WEB UI] Fix log page buttons for reverse proxy mode
  • [SPARK-20190][APP-ID] applications//jobs’ in rest api,status should be [running|succeeded|failed|unknown]
  • [SPARK-20197][SPARKR][BRANCH-2.1] CRAN check fail with package installation
  • [SPARK-19999][BACKPORT-2.1][CORE] Workaround JDK-8165231 to identify PPC64 architectures as supporting unaligned access
  • [SPARK-20084][CORE] Remove internal.metrics.updatedBlockStatuses from history files.
  • [SPARK-20164][SQL] AnalysisException not tolerant of null query plan.
  • [SPARK-13446][BACKPORT][SQL] Support reading data from Hive 2.0.1 metastore
  • [SPARK-20134][SQL] SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates
  • [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini
  • [SPARK-14536][SQL][BACKPORT-2.1] fix to handle null value in array type column for postgres.
  • [SPARK-20125][SQL] Dataset of type option of map does not work
  • [SPARK-20102] Fix nightly packaging and RC packaging scripts w/ two minor build fixes
  • [SPARK-20086][SQL] CollapseWindow should not collapse dependent adjacent windows
  • [SPARK-19674][SQL] Ignore driver accumulator updates don’t belong to …
  • [SPARK-19959][SQL] Fix to throw NullPointerException in df[java.lang.Long].collect
  • [SPARK-20070][BRANCH-2.1] Redact DataSourceScanExec treeString
  • [SPARK-19970][SQL][BRANCH-2.1] Table owner should be USER instead of PRINCIPAL in kerberized clusters
  • [SPARK-20021][PYSPARK] Miss backslash in python code
  • [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fails when it was called on executors.
  • [SPARK-19980][SQL][BACKPORT-2.1] Add NULL checks in Bean serializer
  • [SPARK-19237][SPARKR][CORE] On Windows spark-submit should handle when java is not installed
  • [SPARK-20017][SQL] change the nullability of function ‘StringToMap’ from ‘false’ to ‘true’
  • [SPARK-19912][SQL] String literals should be escaped for Hive metastore partition pruning
  • [SPARK-17204][CORE] Fix replicated off heap storage
  • [SPARK-19994][SQL] Wrong outputOrdering for right/full outer smj
  • [SPARK-18817][SPARKR][SQL] change derby log output to temp dir
  • [SPARK-19721][SS][BRANCH-2.1] Good error message for version mismatch in log files
  • [SPARK-19765][SPARK-18549][SPARK-19093][SPARK-19736][BACKPORT-2.1][SQL] Backport Three Cache-related PRs to Spark 2.1
  • [SPARK-19329][SQL][BRANCH-2.1] Reading from or writing to a datasource table with a non pre-existing location should succeed
  • [SPARK-19872] [PYTHON] Use the correct deserializer for RDD construction for coalesce/repartition
  • [SPARK-19944][SQL] Move SQLConf from sql/core to sql/catalyst (branch-2.1)
  • [SPARK-19887][SQL] dynamic partition keys can be null or empty string
  • [SPARK-19933][SQL] Do not change output of a subquery
  • [SPARK-19853][SS] uppercase kafka topics fail when startingOffsets are SpecificOffsets
  • [SPARK-19611][SQL] Introduce configurable table schema inference
  • [SPARK-19893][SQL] should not run DataFrame set oprations with map type
  • [SPARK-19891][SS] Await Batch Lock notified on stream execution exit
  • [SPARK-19886] Fix reportDataLoss if statement in SS KafkaSource
  • [SPARK-19861][SS] watermark should not be a negative time.
  • [SPARK-19561][SQL] add int case handling for TimestampType
  • [SPARK-19859][SS][FOLLOW-UP] The new watermark should override the old one.
  • Revert “[SPARK-19413][SS] MapGroupsWithState for arbitrary stateful operations for branch-2.1”
  • [SPARK-19813] maxFilesPerTrigger combo latestFirst may miss old files in combination with maxFileAge in FileStreamSource
  • [SPARK-18055][SQL] Use correct mirror in ExpresionEncoder
  • [SPARK-19348][PYTHON] PySpark keyword_only decorator is not thread-safe
  • [SPARK-19859][SS] The new watermark should override the old one
  • Revert “[SPARK-19561] [PYTHON] cast TimestampType.toInternal output to long”
  • [SPARK-19561] [PYTHON] cast TimestampType.toInternal output to long
  • [SPARK-19719][SS] Kafka writer for both structured streaming and batch queires
  • [SPARK-19774] StreamExecution should call stop() on sources when a stream fails
  • [SPARK-19779][SS] Delete needless tmp file after restart structured streaming job
  • [SPARK-19750][UI][BRANCH-2.1] Fix redirect issue from http to https
  • [SPARK-19373][MESOS] Base spark.scheduler.minRegisteredResourceRatio …
  • [SPARK-19766][SQL] Constant alias columns in INNER JOIN should not be folded by FoldablePropagation rule
  • [SPARK-19572][SPARKR] Allow to disable hive in sparkR shell
  • [SPARK-19677][SS] Committing a delta file atop an existing one should not fail on HDFS
  • [SPARK-19748][SQL] refresh function has a wrong order to do cache invalidate and regenerate the inmemory var for InMemoryFileIndex with FileStatusCache
  • [SPARK-19594][STRUCTURED STREAMING] StreamingQueryListener fails to handle QueryTerminatedEvent if more then one listeners exists
  • [SPARK-14772][PYTHON][ML] Fixed Params.copy method to match Scala implementation
  • [SPARK-19707][CORE] Improve the invalid path check for sc.addJar
  • [SPARK-19691][SQL][BRANCH-2.1] Fix ClassCastException when calculating percentile of decimal column
  • [SPARK-19459][SQL][BRANCH-2.1] Support for nested char/varchar fields in ORC

Known Issues

  • Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

Maintenance Updates

Maintenance updates made to the 2.1.1-db6 cluster image since its initial release include:

  • May 31, 2018
    • Fixed a bug affecting Spark SQL execution engine.
  • Mar 30, 2018
    • Fixed an issue caused by a race condition that could, in rare circumstances, lead to loss of some output files.

System Environment

  • Operating System: Ubuntu 16.04.1 LTS
  • Java: 1.8.0_111
  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
  • Python: 2.7.12 (or 3.5.2 if Python 3 support is enabled)
  • R: R version 3.2.3 (2015-12-10)

Pre-installed Python Libraries

Library Version Library Version Library Version
ansi2html 1.1.1 argparse 1.2.1 boto 2.42.0
boto3 1.4.1 botocore 1.4.70 brewer2mpl 1.4.1
certifi 2016.2.28 cffi 1.7.0 chardet 2.3.0
colorama 0.3.7 configobj 5.0.6 cryptography 1.5
cycler 0.10.0 Cython 0.24.1 decorator 4.0.10
docutils 0.13.1 enum34 1.1.6 et-xmlfile 1.0.1
freetype-py 1.0.2 funcsigs 1.0.2 fusepy 2.0.4
futures 3.0.5 ggplot 0.6.8 html5lib 0.999
idna 2.1 ipaddress 1.0.16 ipython 2.2.0
ipython-genutils 0.1.0 jdcal 1.2 Jinja2 2.8
jmespath 0.9.0 llvmlite 0.13.0 lxml 3.6.4
MarkupSafe 0.23 matplotlib 1.5.3 mpld3 0.2
msgpack-python 0.4.7 ndg-httpsclient 0.3.3 numba 0.28.1
numpy 1.11.1 openpyxl 2.3.2 pandas 0.18.1
pathlib2 2.1.0 patsy 0.4.1 pexpect 4.0.1
pickleshare 0.7.4 Pillow 3.3.1 pip 9.0.1
ply 3.9 prompt-toolkit 1.0.7 psycopg2 2.6.2
ptyprocess 0.5.1 py4j 0.10.3 pyasn1 0.1.9
pycparser 2.14 Pygments 2.1.3 PyGObject 3.20.0
pyOpenSSL 16.0.0 pyparsing 2.1.4 pypng 0.0.18
Python 2.7.12 python-dateutil 2.5.3 python-geohash 0.8.5
pytz 2016.6.1 requests 2.11.1 s3transfer 0.1.9
scikit-learn 0.17.1 scipy 0.18.1 scour 0.32
seaborn 0.7.1 setuptools 32.3.1 simplejson 3.8.2
simples3 1.0 singledispatch 3.4.0.3 six 1.10.0
statsmodels 0.6.1 traitlets 4.3.0 urllib3 1.19.1
virtualenv 15.0.1 wcwidth 0.1.7 wheel 0.30.0a0
wsgiref 0.1.2        

Pre-installed R Libraries

Library Version Library Version Library Version
abind 1.4-3 assertthat 0.1 base 3.2.3
BH 1.60.0-2 bitops 1.0-6 boot 1.3-17
brew 1.0-6 car 2.1-3 caret 6.0-71
chron 2.3-47 class 7.3-14 cluster 2.0.5
codetools 0.2-14 colorspace 1.2-4 compiler 3.2.3
crayon 1.3.1 curl 2.2 data.table 1.9.6
datasets 3.2.3 DBI 0.5-1 devtools 1.12.0
dichromat 2.0-0 digest 0.6.9 doMC 1.3.4
dplyr 0.5.0 foreach 1.4.3 foreign 0.8-66
gbm 2.1.1 ggplot2 2.1.0 git2r 0.15.0
glmnet 2.0-5 graphics 3.2.3 grDevices 3.2.3
grid 3.2.3 gsubfn 0.6-6 gtable 0.1.2
h2o 3.10.0.8 httr 1.2.1 hwriter 1.3.2
hwriterPlus 1.0-3 iterators 1.0.8 jsonlite 1.1
KernSmooth 2.23-15 labeling 0.3 lattice 0.20-34
lazyeval 0.2.0 littler 0.3.0 lme4 1.1-12
lubridate 1.6.0 magrittr 1.5 mapproj 1.2-4
maps 3.0.2 MASS 7.3-45 Matrix 1.2-7.1
MatrixModels 0.4-1 memoise 1.0.0 methods 3.2.3
mgcv 1.8-11 mime 0.5 minqa 1.2.4
multicore 0.2 munsell 0.4.2 mvtnorm 1.0-5
nlme 3.1-124 nloptr 1.0.4 nnet 7.3-12
openssl 0.9.4 parallel 3.2.3 pbkrtest 0.4-6
pkgKitten 0.1.3 plyr 1.8.4 praise 1.0.0
pROC 1.8 proto 0.3-10 quantreg 5.29
R.methodsS3 1.7.1 R.oo 1.20.0 R.utils 2.4.0
R6 2.2.0 randomForest 4.6-12 RColorBrewer 1.1-2
Rcpp 0.12.7 RcppEigen 0.3.2.9.0 RCurl 1.95-4.8
reshape2 1.4.2 RODBC 1.3-12 roxygen2 5.0.1
rpart 4.1-10 Rserve 1.7-3 RSQLite 1.0.0
rstudioapi 0.6 scales 0.3.0 sp 1.0-15
SparkR 2.1.1 SparseM 1.72 spatial 7.3-11
splines 3.2.3 sqldf 0.4-10 statmod 1.4.26
stats 3.2.3 stats4 3.2.3 stringi 1.0-1
stringr 1.0.0 survival 2.38-3 tcltk 3.2.3
TeachingDemos 2.10 testthat 1.0.2 tibble 1.2
tools 3.2.3 utils 3.2.3 whisker 0.3-2
withr 1.0.2        

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws aws-java-sdk 1.9.40
com.amazonaws aws-java-sdk-autoscaling 1.9.40
com.amazonaws aws-java-sdk-cloudformation 1.9.40
com.amazonaws aws-java-sdk-cloudfront 1.9.40
com.amazonaws aws-java-sdk-cloudhsm 1.9.40
com.amazonaws aws-java-sdk-cloudsearch 1.9.40
com.amazonaws aws-java-sdk-cloudtrail 1.9.40
com.amazonaws aws-java-sdk-cloudwatch 1.9.40
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.9.40
com.amazonaws aws-java-sdk-codedeploy 1.9.40
com.amazonaws aws-java-sdk-cognitoidentity 1.9.40
com.amazonaws aws-java-sdk-cognitosync 1.9.40
com.amazonaws aws-java-sdk-config 1.9.40
com.amazonaws aws-java-sdk-core 1.9.40
com.amazonaws aws-java-sdk-datapipeline 1.9.40
com.amazonaws aws-java-sdk-directconnect 1.9.40
com.amazonaws aws-java-sdk-directory 1.9.40
com.amazonaws aws-java-sdk-dynamodb 1.9.40
com.amazonaws aws-java-sdk-ec2 1.9.40
com.amazonaws aws-java-sdk-ecs 1.9.40
com.amazonaws aws-java-sdk-efs 1.9.40
com.amazonaws aws-java-sdk-elasticache 1.9.40
com.amazonaws aws-java-sdk-elasticbeanstalk 1.9.40
com.amazonaws aws-java-sdk-elasticloadbalancing 1.9.40
com.amazonaws aws-java-sdk-elastictranscoder 1.9.40
com.amazonaws aws-java-sdk-emr 1.9.40
com.amazonaws aws-java-sdk-glacier 1.9.40
com.amazonaws aws-java-sdk-iam 1.9.40
com.amazonaws aws-java-sdk-importexport 1.9.40
com.amazonaws aws-java-sdk-kinesis 1.9.40
com.amazonaws aws-java-sdk-kms 1.9.40
com.amazonaws aws-java-sdk-lambda 1.9.40
com.amazonaws aws-java-sdk-logs 1.9.40
com.amazonaws aws-java-sdk-machinelearning 1.9.40
com.amazonaws aws-java-sdk-opsworks 1.9.40
com.amazonaws aws-java-sdk-rds 1.9.40
com.amazonaws aws-java-sdk-redshift 1.9.40
com.amazonaws aws-java-sdk-route53 1.9.40
com.amazonaws aws-java-sdk-s3 1.9.40
com.amazonaws aws-java-sdk-ses 1.9.40
com.amazonaws aws-java-sdk-simpledb 1.9.40
com.amazonaws aws-java-sdk-simpleworkflow 1.9.40
com.amazonaws aws-java-sdk-sns 1.9.40
com.amazonaws aws-java-sdk-sqs 1.9.40
com.amazonaws aws-java-sdk-ssm 1.9.40
com.amazonaws aws-java-sdk-storagegateway 1.9.40
com.amazonaws aws-java-sdk-sts 1.9.40
com.amazonaws aws-java-sdk-support 1.9.40
com.amazonaws aws-java-sdk-swf-libraries 1.9.40
com.amazonaws aws-java-sdk-workspaces 1.9.40
com.chuusai shapeless_2.10.4 2.0.0
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.10 0.1.2-spark2.1
com.databricks dbml-local_2.10-tests 0.1.2-spark2.1
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.10 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.10 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.4.5
com.fasterxml.jackson.core jackson-core 2.4.5
com.fasterxml.jackson.core jackson-databind 2.4.5
com.fasterxml.jackson.datatype jackson-datatype-joda 2.4.5
com.fasterxml.jackson.module jackson-module-paranamer 2.4.5
com.fasterxml.jackson.module jackson-module-scala_2.10 2.4.5
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.10 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.10 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.10 6.23.0
com.twitter util-core_2.10 6.23.0
com.twitter util-jvm_2.10 6.23.0
com.typesafe config 1.2.1
com.typesafe scalalogging-slf4j_2.10 1.1.0
com.univocity univocity-parsers 2.2.1
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.8.0.Final
io.netty netty-all 4.0.42.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.java.dev.jets3t jets3t 0.7.1
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.py4j py4j 0.10.4
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.1
org.apache.parquet parquet-common 1.8.1
org.apache.parquet parquet-encoding 1.8.1
org.apache.parquet parquet-format 2.3.0-incubating
org.apache.parquet parquet-hadoop 1.8.1
org.apache.parquet parquet-jackson 1.8.1
org.apache.thrift libfb303 0.9.2
org.apache.thrift libthrift 0.9.2
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.3.v20150827
org.eclipse.jetty jetty-continuation 9.3.3.v20150827
org.eclipse.jetty jetty-http 9.3.3.v20150827
org.eclipse.jetty jetty-io 9.3.3.v20150827
org.eclipse.jetty jetty-jndi 9.3.3.v20150827
org.eclipse.jetty jetty-plus 9.3.3.v20150827
org.eclipse.jetty jetty-proxy 9.3.3.v20150827
org.eclipse.jetty jetty-security 9.3.3.v20150827
org.eclipse.jetty jetty-server 9.3.3.v20150827
org.eclipse.jetty jetty-servlet 9.3.3.v20150827
org.eclipse.jetty jetty-servlets 9.3.3.v20150827
org.eclipse.jetty jetty-util 9.3.3.v20150827
org.eclipse.jetty jetty-webapp 9.3.3.v20150827
org.eclipse.jetty jetty-xml 9.3.3.v20150827
org.fusesource.jansi jansi 1.4
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.10 3.2.11
org.json4s json4s-core_2.10 3.2.11
org.json4s json4s-jackson_2.10 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rosuda.REngine REngine 2.1.0
org.scala-lang jline 2.10.6
org.scala-lang scala-compiler_2.10 2.10.6
org.scala-lang scala-library_2.10 2.10.6
org.scala-lang scala-reflect_2.10 2.10.6
org.scala-lang scalap_2.10 2.10.6
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.10 1.12.5
org.scalamacros quasiquotes_2.10 2.0.0
org.scalanlp breeze-macros_2.10 0.12
org.scalanlp breeze_2.10 0.12
org.scalatest scalatest_2.10 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.10 0.7.4
org.spire-math spire_2.10 0.7.4
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
stax stax-api 1.0.1
xerces xercesImpl 2.9.1
xml-apis xml-apis 1.3.04
xmlenc xmlenc 0.52

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws aws-java-sdk 1.9.40
com.amazonaws aws-java-sdk-autoscaling 1.9.40
com.amazonaws aws-java-sdk-cloudformation 1.9.40
com.amazonaws aws-java-sdk-cloudfront 1.9.40
com.amazonaws aws-java-sdk-cloudhsm 1.9.40
com.amazonaws aws-java-sdk-cloudsearch 1.9.40
com.amazonaws aws-java-sdk-cloudtrail 1.9.40
com.amazonaws aws-java-sdk-cloudwatch 1.9.40
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.9.40
com.amazonaws aws-java-sdk-codedeploy 1.9.40
com.amazonaws aws-java-sdk-cognitoidentity 1.9.40
com.amazonaws aws-java-sdk-cognitosync 1.9.40
com.amazonaws aws-java-sdk-config 1.9.40
com.amazonaws aws-java-sdk-core 1.9.40
com.amazonaws aws-java-sdk-datapipeline 1.9.40
com.amazonaws aws-java-sdk-directconnect 1.9.40
com.amazonaws aws-java-sdk-directory 1.9.40
com.amazonaws aws-java-sdk-dynamodb 1.9.40
com.amazonaws aws-java-sdk-ec2 1.9.40
com.amazonaws aws-java-sdk-ecs 1.9.40
com.amazonaws aws-java-sdk-efs 1.9.40
com.amazonaws aws-java-sdk-elasticache 1.9.40
com.amazonaws aws-java-sdk-elasticbeanstalk 1.9.40
com.amazonaws aws-java-sdk-elasticloadbalancing 1.9.40
com.amazonaws aws-java-sdk-elastictranscoder 1.9.40
com.amazonaws aws-java-sdk-emr 1.9.40
com.amazonaws aws-java-sdk-glacier 1.9.40
com.amazonaws aws-java-sdk-iam 1.9.40
com.amazonaws aws-java-sdk-importexport 1.9.40
com.amazonaws aws-java-sdk-kinesis 1.9.40
com.amazonaws aws-java-sdk-kms 1.9.40
com.amazonaws aws-java-sdk-lambda 1.9.40
com.amazonaws aws-java-sdk-logs 1.9.40
com.amazonaws aws-java-sdk-machinelearning 1.9.40
com.amazonaws aws-java-sdk-opsworks 1.9.40
com.amazonaws aws-java-sdk-rds 1.9.40
com.amazonaws aws-java-sdk-redshift 1.9.40
com.amazonaws aws-java-sdk-route53 1.9.40
com.amazonaws aws-java-sdk-s3 1.9.40
com.amazonaws aws-java-sdk-ses 1.9.40
com.amazonaws aws-java-sdk-simpledb 1.9.40
com.amazonaws aws-java-sdk-simpleworkflow 1.9.40
com.amazonaws aws-java-sdk-sns 1.9.40
com.amazonaws aws-java-sdk-sqs 1.9.40
com.amazonaws aws-java-sdk-ssm 1.9.40
com.amazonaws aws-java-sdk-storagegateway 1.9.40
com.amazonaws aws-java-sdk-sts 1.9.40
com.amazonaws aws-java-sdk-support 1.9.40
com.amazonaws aws-java-sdk-swf-libraries 1.9.40
com.amazonaws aws-java-sdk-workspaces 1.9.40
com.chuusai shapeless_2.11 2.0.0
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.11 0.1.2-spark2.1
com.databricks dbml-local_2.11-tests 0.1.2-spark2.1
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.11 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.11 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.4.5
com.fasterxml.jackson.core jackson-core 2.4.5
com.fasterxml.jackson.core jackson-databind 2.4.5
com.fasterxml.jackson.datatype jackson-datatype-joda 2.4.5
com.fasterxml.jackson.module jackson-module-paranamer 2.4.5
com.fasterxml.jackson.module jackson-module-scala_2.11 2.4.5
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.11 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.11 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.11 6.23.0
com.twitter util-core_2.11 6.23.0
com.twitter util-jvm_2.11 6.23.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging-api_2.11 2.1.2
com.typesafe.scala-logging scala-logging-slf4j_2.11 2.1.2
com.univocity univocity-parsers 2.2.1
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.8.0.Final
io.netty netty-all 4.0.42.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.java.dev.jets3t jets3t 0.7.1
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.py4j py4j 0.10.4
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.1
org.apache.parquet parquet-common 1.8.1
org.apache.parquet parquet-encoding 1.8.1
org.apache.parquet parquet-format 2.3.0-incubating
org.apache.parquet parquet-hadoop 1.8.1
org.apache.parquet parquet-jackson 1.8.1
org.apache.thrift libfb303 0.9.2
org.apache.thrift libthrift 0.9.2
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.3.v20150827
org.eclipse.jetty jetty-continuation 9.3.3.v20150827
org.eclipse.jetty jetty-http 9.3.3.v20150827
org.eclipse.jetty jetty-io 9.3.3.v20150827
org.eclipse.jetty jetty-jndi 9.3.3.v20150827
org.eclipse.jetty jetty-plus 9.3.3.v20150827
org.eclipse.jetty jetty-proxy 9.3.3.v20150827
org.eclipse.jetty jetty-security 9.3.3.v20150827
org.eclipse.jetty jetty-server 9.3.3.v20150827
org.eclipse.jetty jetty-servlet 9.3.3.v20150827
org.eclipse.jetty jetty-servlets 9.3.3.v20150827
org.eclipse.jetty jetty-util 9.3.3.v20150827
org.eclipse.jetty jetty-webapp 9.3.3.v20150827
org.eclipse.jetty jetty-xml 9.3.3.v20150827
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.11 3.2.11
org.json4s json4s-core_2.11 3.2.11
org.json4s json4s-jackson_2.11 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.11 2.11.8
org.scala-lang scala-library_2.11 2.11.8
org.scala-lang scala-reflect_2.11 2.11.8
org.scala-lang scalap_2.11 2.11.8
org.scala-lang.modules scala-parser-combinators_2.11 1.0.2
org.scala-lang.modules scala-xml_2.11 1.0.2
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.11 1.12.5
org.scalanlp breeze-macros_2.11 0.12
org.scalanlp breeze_2.11 0.12
org.scalatest scalatest_2.11 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.11 0.7.4
org.spire-math spire_2.11 0.7.4
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
stax stax-api 1.0.1
xerces xercesImpl 2.9.1
xml-apis xml-apis 1.3.04
xmlenc xmlenc 0.52