2.1.0-db2 Cluster Image

Databricks released this image in late January, 2017.

Important

This release has been deprecated. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks Runtime Versioning and Deprecation Policy.

The following release notes provide information about the Spark 2.1.0-db2 cluster image powered by Apache Spark.

Changes and Improvements

  • The Avro and Amazon Redshift data sources are now bundled in the cluster image. Please see the linked documentation on these data sources for more details.
  • Added Kafka 0.8 support for Structured Streaming. You can use format("kafka08") to read data from a Kafka 0.8+ cluster.
  • Added minPartitions option to Kafka 0.10 Streaming DataSource to configure parallelism when reading from Kafka.
  • Fixed a bug in the Amazon Redshift data source where the combination of filter pushdown and zero column selection (e.g. count(*) queries) caused runtime errors.
  • Changed the default file system for the s3a:// URL scheme to S3AFileSystem instead of DBFS.

Apache Spark

2.1.0-db2 cluster image includes the Apache Spark 2.1.0 release. You can consult JIRA for the detailed changes. 2.1.0-db2 cluster image also includes the following extra bug fixes and improvements:

  • [SPARK-4105][BACKPORT] retry the fetch or stage if shuffle block is corrupt
  • [SPARK-18917] Remove schema check in appending data
  • [SPARK-19314][SS][CATALYST] Do not allow sort before aggregation in Structured Streaming plan
  • [SPARK-19295][SQL] IsolatedClientLoader’s downloadVersion should log the location of downloaded metastore client jars
  • [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor the error checking when append data to an existing table
  • [SPARK-18475] Be able to increase parallelism in StructuredStreaming Kafka source
  • [SPARK-19168][STRUCTURED STREAMING] StateStore should be aborted upon error
  • [SPARK-19113][SS][TESTS] Ignore StreamingQueryException thrown from awaitInitialization to avoid breaking tests
  • [SPARK-19231][SPARKR] add error handling for download and untar for Spark release
  • [SPARK-19066][SPARKR][BACKPORT-2.1] LDA doesn’t set optimizer correctly
  • [SPARK-17755][CORE] Use workerRef to send RegisterWorkerResponse to avoid the race condition
  • [SPARK-19129][SQL] SessionCatalog: Disallow empty part col values in partition spec
  • [SPARK-19065][SQL] Don’t inherit expression id in dropDuplicates
  • [SPARK-19019][PYTHON] Fix hijacked collections.namedtuple and port cloudpickle changes for PySpark to work with Python 3.6.0
  • [SPARK-18905][STREAMING] Fix the issue of removing a failed jobset from JobScheduler.jobSets
  • [SPARK-19232][SPARKR] Update Spark distribution download cache location on Windows
  • [SPARK-19082][SQL] Make ignoreCorruptFiles work for Parquet
  • [SPARK-19092][SQL][BACKPORT-2.1] Save() API of DataFrameWriter should not scan all the saved files #16481
  • [SPARK-19120] Refresh Metadata Cache After Loading Hive Tables
  • [SPARK-19180][SQL] the offset of short should be 2 in OffHeapColumn
  • [SPARK-18335][SPARKR] createDataFrame to support numPartitions parameter
  • [SPARK-19178][SQL] convert string of large numbers to int should return null
  • [SPARK-18687][PYSPARK][SQL] Backward compatibility - creating a Dataframe on a new SQLContext object fails with a Derby error
  • [SPARK-17237][SQL] Remove backticks in a pivot result schema
  • [SPARK-19055][SQL][PYSPARK] Fix SparkSession initialization when SparkContext is stopped
  • [SPARK-18969][SQL] Support grouping by nondeterministic expressions
  • [SPARK-18857][SQL] Don’t use Iterator.duplicate for incrementalCollect in Thrift Server
  • [SPARK-19158][SPARKR][EXAMPLES] Fix ml.R example fails due to lack of e1071 package.
  • [SPARK-19130][SPARKR] Support setting literal value as column implicitly
  • [SPARK-19133][SPARKR][ML][BACKPORT-2.1] fix glm for Gamma, clarify glm family supported
  • [SPARK-19140][SS] Allow update mode for non-aggregation streaming queries
  • [SPARK-18997][CORE] Recommended upgrade libthrift to 0.9.3
  • [SPARK-19113][SS][TESTS] Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization
  • [SPARK-19137][SQL] Fix withSQLConf to reset OptionalConfigEntry correctly
  • [SPARK-16845][SQL] GeneratedClass$SpecificOrdering grows beyond 64 KB
  • [SPARK-18952][BACKPORT] Regex strings not properly escaped in codegen for aggregations
  • [SPARK-18903][SPARKR][BACKPORT-2.1] Add API to get SparkUI URL
  • [SPARK-19126][DOCS] Update Join Documentation Across Languages
  • [SPARK-19127][DOCS] Update Rank Function Documentation
  • [SPARK-18941][SQL][DOC] Add a new behavior document on CREATE/DROP TABLE with LOCATION
  • [SPARK-19106][DOCS] Styling for the configuration docs is broken
  • [SPARK-19110][ML][MLLIB] DistributedLDAModel returns different logPrior for original and loaded model
  • [SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options
  • [SPARK-19083] sbin/start-history-server.sh script use of $@ without quotes
  • [SPARK-19033][CORE] Add admin acls for history server
  • [SPARK-18877][SQL][BACKPORT-2.1] CSVInferSchema.inferField on DecimalType should find a common type with typeSoFar
  • [SPARK-19048][SQL] Delete Partition Location when Dropping Managed Partitioned Tables in InMemoryCatalog
  • [SPARK-19028][SQL] Fixed non-thread-safe functions used in SessionCatalog
  • [SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide
  • [SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.
  • [SPARK-19050][SS][TESTS] Fix EventTimeWatermarkSuite ‘delay in months and years handled correctly’
  • [SPARK-19016][SQL][DOC] Document scalable partition handling
  • [SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD
  • [SPARK-18775][SQL] Limit the max number of records written per file
  • [SPARK-17910][SQL] Allow users to update the comment of a column
  • [SPARK-18669][SS][DOCS] Update Apache docs for Structured Streaming regarding watermarking and status
  • [SPARK-18993][BUILD] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags
  • [SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI
  • [SPARK-18991][CORE] Change ContextCleaner.referenceBuffer to use ConcurrentHashMap to make it faster
  • [SPARK-18972][CORE] Fix the netty thread names for RPC
  • [SPARK-18985][SS] Add missing @InterfaceStability.Evolving for Structured Streaming APIs
  • [SPARK-17807][CORE] split test-tags into test-JAR
  • [SPARK-18973][SQL] Remove SortPartitions and RedistributeData
  • [SPARK-18908][SS] Creating StreamingQueryException should check if logicalPlan is created
  • [SPARK-18528][SQL] Fix a bug to initialise an iterator of aggregation buffer
  • [SPARK-18234][SS] Made update mode public
  • [SPARK-18588][SS][KAFKA] Create a new KafkaConsumer when error happens to fix the flaky test
  • [SPARK-18949][SQL][BACKPORT-2.1] Add recoverPartitions API to Catalog
  • [SPARK-18954][TESTS] Fix flaky test: o.a.s.streaming.BasicOperationsSuite rdd cleanup - map and window
  • [SPARK-18031][TESTS] Fix flaky test ExecutorAllocationManagerSuite.basic functionality
  • [SPARK-18894][SS] Fix event time watermark delay threshold specified in months or years
  • [SPARK-18947][SQL] SQLContext.tableNames should not call Catalog.listTables
  • [SPARK-18900][FLAKY-TEST] StateStoreSuite.maintenance
  • [SPARK-18927][SS] MemorySink for StructuredStreaming can’t recover from checkpoint if location is provided in SessionConf
  • [SPARK-18281][SQL] [PYSPARK] Remove timeout for reading data through socket for local iterator
  • [SPARK-18761][CORE] Introduce “task reaper” to oversee task killing in executors
  • [SPARK-18928] Check TaskContext.isInterrupted() in FileScanRDD, JDBCRDD & UnsafeSorter
  • [SPARK-18921][SQL] check database existence with Hive.databaseExists instead of getDatabase
  • [SPARK-18700][SQL] Add StripedLock for each table’s relation in cache
  • [SPARK-18703][SPARK-18675][SQL][BACKPORT-2.1] CTAS for hive serde table should work for all hive versions AND Drop Staging Directories and Data Files
  • [SPARK-18827][CORE] Fix cannot read broadcast on disk
  • [SPARK-18918][DOC] Missing </td> in Configuration page
  • [SPARK-18849][ML][SPARKR][DOC] vignettes final check reorg
  • [SPARK-18186][BRANCH-2.1] Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support
  • [SPARK-18904][SS][TESTS] Merge two FileStreamSourceSuite files
  • [SPARK-18897][SPARKR] Fix SparkR SQL Test to drop test table
  • [SPARK-18108][SQL] Fix a schema inconsistent bug that makes a parquet reader fail to read data
  • [SPARK-18850][SS] Make StreamExecution and progress classes serializable
  • [SPARK-18892][SQL] Alias percentile_approx approx_percentile

Known Issues

  • Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

System Environment

  • Operating System: Ubuntu 16.04.1 LTS
  • Java: 1.8.0_111
  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
  • Python: 2.7.12 (or 3.5.2 if Python 3 support is enabled)
  • R: R version 3.2.3 (2015-12-10)

Pre-installed Python Libraries

Library Version Library Version Library Version
ansi2html 1.1.1 argparse 1.2.1 boto 2.42.0
boto3 1.4.1 botocore 1.4.70 brewer2mpl 1.4.1
certifi 2016.2.28 cffi 1.7.0 chardet 2.3.0
colorama 0.3.7 configobj 5.0.6 cryptography 1.5
cycler 0.10.0 Cython 0.24.1 decorator 4.0.10
docutils 0.13.1 enum34 1.1.6 et-xmlfile 1.0.1
freetype-py 1.0.2 funcsigs 1.0.2 fusepy 2.0.4
futures 3.0.5 ggplot 0.6.8 html5lib 0.999
idna 2.1 ipaddress 1.0.16 ipython 2.2.0
ipython-genutils 0.1.0 jdcal 1.2 Jinja2 2.8
jmespath 0.9.0 llvmlite 0.13.0 lxml 3.6.4
MarkupSafe 0.23 matplotlib 1.5.3 mpld3 0.2
msgpack-python 0.4.7 ndg-httpsclient 0.3.3 numba 0.28.1
numpy 1.11.1 openpyxl 2.3.2 pandas 0.18.1
pathlib2 2.1.0 patsy 0.4.1 pexpect 4.0.1
pickleshare 0.7.4 Pillow 3.3.1 pip 9.0.1
pkg_resources 0.0.0 ply 3.9 prompt-toolkit 1.0.7
psycopg2 2.6.2 ptyprocess 0.5.1 py4j 0.10.3
pyasn1 0.1.9 pycparser 2.14 Pygments 2.1.3
PyGObject 3.20.0 pyOpenSSL 16.0.0 pyparsing 2.1.4
pypng 0.0.18 Python 2.7.12 python-dateutil 2.5.3
python-geohash 0.8.5 pytz 2016.6.1 requests 2.11.1
s3transfer 0.1.9 scikit-learn 0.17.1 scipy 0.18.1
scour 0.32 seaborn 0.7.1 setuptools 32.3.1
simplejson 3.8.2 simples3 1.0 singledispatch 3.4.0.3
six 1.10.0 statsmodels 0.6.1 traitlets 4.3.0
urllib3 1.19.1 virtualenv 15.0.1 wcwidth 0.1.7
wheel 0.30.0a0 wsgiref 0.1.2    

Pre-installed R Libraries

Library Version Library Version Library Version
abind 1.4-3 assertthat 0.1 base 3.2.3
BH 1.60.0-2 bitops 1.0-6 boot 1.3-17
brew 1.0-6 car 2.1-3 caret 6.0-71
chron 2.3-47 class 7.3-14 cluster 2.0.5
codetools 0.2-14 colorspace 1.2-4 compiler 3.2.3
crayon 1.3.1 curl 2.2 data.table 1.9.6
datasets 3.2.3 DBI 0.5-1 devtools 1.12.0
dichromat 2.0-0 digest 0.6.9 doMC 1.3.4
dplyr 0.5.0 foreach 1.4.3 foreign 0.8-66
gbm 2.1.1 ggplot2 2.1.0 git2r 0.15.0
glmnet 2.0-5 graphics 3.2.3 grDevices 3.2.3
grid 3.2.3 gsubfn 0.6-6 gtable 0.1.2
h2o 3.10.0.8 httr 1.2.1 hwriter 1.3.2
hwriterPlus 1.0-3 iterators 1.0.8 jsonlite 1.1
KernSmooth 2.23-15 labeling 0.3 lattice 0.20-34
lazyeval 0.2.0 littler 0.3.0 lme4 1.1-12
lubridate 1.6.0 magrittr 1.5 mapproj 1.2-4
maps 3.0.2 MASS 7.3-45 Matrix 1.2-7.1
MatrixModels 0.4-1 memoise 1.0.0 methods 3.2.3
mgcv 1.8-11 mime 0.5 minqa 1.2.4
multicore 0.2 munsell 0.4.2 mvtnorm 1.0-5
nlme 3.1-124 nloptr 1.0.4 nnet 7.3-12
openssl 0.9.4 parallel 3.2.3 pbkrtest 0.4-6
pkgKitten 0.1.3 plyr 1.8.4 praise 1.0.0
pROC 1.8 proto 0.3-10 quantreg 5.29
R.methodsS3 1.7.1 R.oo 1.20.0 R.utils 2.4.0
R6 2.2.0 randomForest 4.6-12 RColorBrewer 1.1-2
Rcpp 0.12.7 RcppEigen 0.3.2.9.0 RCurl 1.95-4.8
reshape2 1.4.2 RODBC 1.3-12 roxygen2 5.0.1
rpart 4.1-10 Rserve 1.7-3 RSQLite 1.0.0
rstudioapi 0.6 scales 0.3.0 sp 1.0-15
SparkR 2.1.0 SparseM 1.72 spatial 7.3-11
splines 3.2.3 sqldf 0.4-10 statmod 1.4.26
stats 3.2.3 stats4 3.2.3 stringi 1.0-1
stringr 1.0.0 survival 2.38-3 tcltk 3.2.3
TeachingDemos 2.10 testthat 1.0.2 tibble 1.2
tools 3.2.3 utils 3.2.3 whisker 0.3-2
withr 1.0.2        

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws aws-java-sdk 1.9.40
com.amazonaws aws-java-sdk-autoscaling 1.9.40
com.amazonaws aws-java-sdk-cloudformation 1.9.40
com.amazonaws aws-java-sdk-cloudfront 1.9.40
com.amazonaws aws-java-sdk-cloudhsm 1.9.40
com.amazonaws aws-java-sdk-cloudsearch 1.9.40
com.amazonaws aws-java-sdk-cloudtrail 1.9.40
com.amazonaws aws-java-sdk-cloudwatch 1.9.40
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.9.40
com.amazonaws aws-java-sdk-codedeploy 1.9.40
com.amazonaws aws-java-sdk-cognitoidentity 1.9.40
com.amazonaws aws-java-sdk-cognitosync 1.9.40
com.amazonaws aws-java-sdk-config 1.9.40
com.amazonaws aws-java-sdk-core 1.9.40
com.amazonaws aws-java-sdk-datapipeline 1.9.40
com.amazonaws aws-java-sdk-directconnect 1.9.40
com.amazonaws aws-java-sdk-directory 1.9.40
com.amazonaws aws-java-sdk-dynamodb 1.9.40
com.amazonaws aws-java-sdk-ec2 1.9.40
com.amazonaws aws-java-sdk-ecs 1.9.40
com.amazonaws aws-java-sdk-efs 1.9.40
com.amazonaws aws-java-sdk-elasticache 1.9.40
com.amazonaws aws-java-sdk-elasticbeanstalk 1.9.40
com.amazonaws aws-java-sdk-elasticloadbalancing 1.9.40
com.amazonaws aws-java-sdk-elastictranscoder 1.9.40
com.amazonaws aws-java-sdk-emr 1.9.40
com.amazonaws aws-java-sdk-glacier 1.9.40
com.amazonaws aws-java-sdk-iam 1.9.40
com.amazonaws aws-java-sdk-importexport 1.9.40
com.amazonaws aws-java-sdk-kinesis 1.9.40
com.amazonaws aws-java-sdk-kms 1.9.40
com.amazonaws aws-java-sdk-lambda 1.9.40
com.amazonaws aws-java-sdk-logs 1.9.40
com.amazonaws aws-java-sdk-machinelearning 1.9.40
com.amazonaws aws-java-sdk-opsworks 1.9.40
com.amazonaws aws-java-sdk-rds 1.9.40
com.amazonaws aws-java-sdk-redshift 1.9.40
com.amazonaws aws-java-sdk-route53 1.9.40
com.amazonaws aws-java-sdk-s3 1.9.40
com.amazonaws aws-java-sdk-ses 1.9.40
com.amazonaws aws-java-sdk-simpledb 1.9.40
com.amazonaws aws-java-sdk-simpleworkflow 1.9.40
com.amazonaws aws-java-sdk-sns 1.9.40
com.amazonaws aws-java-sdk-sqs 1.9.40
com.amazonaws aws-java-sdk-ssm 1.9.40
com.amazonaws aws-java-sdk-storagegateway 1.9.40
com.amazonaws aws-java-sdk-sts 1.9.40
com.amazonaws aws-java-sdk-support 1.9.40
com.amazonaws aws-java-sdk-swf-libraries 1.9.40
com.amazonaws aws-java-sdk-workspaces 1.9.40
com.chuusai shapeless_2.10.4 2.0.0
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.10 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.10 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.4.5
com.fasterxml.jackson.core jackson-core 2.4.5
com.fasterxml.jackson.core jackson-databind 2.4.5
com.fasterxml.jackson.datatype jackson-datatype-joda 2.4.5
com.fasterxml.jackson.module jackson-module-paranamer 2.4.5
com.fasterxml.jackson.module jackson-module-scala_2.10 2.4.5
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.10 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.10 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.10 6.23.0
com.twitter util-core_2.10 6.23.0
com.twitter util-jvm_2.10 6.23.0
com.typesafe config 1.2.1
com.typesafe scalalogging-slf4j_2.10 1.1.0
com.univocity univocity-parsers 2.2.1
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.8.0.Final
io.netty netty-all 4.0.42.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.java.dev.jets3t jets3t 0.7.1
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.py4j py4j 0.10.4
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-mapred 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.1
org.apache.parquet parquet-common 1.8.1
org.apache.parquet parquet-encoding 1.8.1
org.apache.parquet parquet-format 2.3.0-incubating
org.apache.parquet parquet-hadoop 1.8.1
org.apache.parquet parquet-jackson 1.8.1
org.apache.thrift libfb303 0.9.2
org.apache.thrift libthrift 0.9.2
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.3.v20150827
org.eclipse.jetty jetty-continuation 9.3.3.v20150827
org.eclipse.jetty jetty-http 9.3.3.v20150827
org.eclipse.jetty jetty-io 9.3.3.v20150827
org.eclipse.jetty jetty-jndi 9.3.3.v20150827
org.eclipse.jetty jetty-plus 9.3.3.v20150827
org.eclipse.jetty jetty-proxy 9.3.3.v20150827
org.eclipse.jetty jetty-security 9.3.3.v20150827
org.eclipse.jetty jetty-server 9.3.3.v20150827
org.eclipse.jetty jetty-servlet 9.3.3.v20150827
org.eclipse.jetty jetty-servlets 9.3.3.v20150827
org.eclipse.jetty jetty-util 9.3.3.v20150827
org.eclipse.jetty jetty-webapp 9.3.3.v20150827
org.eclipse.jetty jetty-xml 9.3.3.v20150827
org.fusesource.jansi jansi 1.4
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.10 3.2.11
org.json4s json4s-core_2.10 3.2.11
org.json4s json4s-jackson_2.10 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rosuda.REngine REngine 2.1.0
org.scala-lang jline 2.10.6
org.scala-lang scala-compiler_2.10 2.10.6
org.scala-lang scala-library_2.10 2.10.6
org.scala-lang scala-reflect_2.10 2.10.6
org.scala-lang scalap_2.10 2.10.6
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.10 1.12.5
org.scalamacros quasiquotes_2.10 2.0.0
org.scalanlp breeze-macros_2.10 0.12
org.scalanlp breeze_2.10 0.12
org.scalatest scalatest_2.10 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.10 0.7.4
org.spire-math spire_2.10 0.7.4
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
stax stax-api 1.0.1
xerces xercesImpl 2.9.1
xml-apis xml-apis 1.3.04
xmlenc xmlenc 0.52

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws aws-java-sdk 1.9.40
com.amazonaws aws-java-sdk-autoscaling 1.9.40
com.amazonaws aws-java-sdk-cloudformation 1.9.40
com.amazonaws aws-java-sdk-cloudfront 1.9.40
com.amazonaws aws-java-sdk-cloudhsm 1.9.40
com.amazonaws aws-java-sdk-cloudsearch 1.9.40
com.amazonaws aws-java-sdk-cloudtrail 1.9.40
com.amazonaws aws-java-sdk-cloudwatch 1.9.40
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.9.40
com.amazonaws aws-java-sdk-codedeploy 1.9.40
com.amazonaws aws-java-sdk-cognitoidentity 1.9.40
com.amazonaws aws-java-sdk-cognitosync 1.9.40
com.amazonaws aws-java-sdk-config 1.9.40
com.amazonaws aws-java-sdk-core 1.9.40
com.amazonaws aws-java-sdk-datapipeline 1.9.40
com.amazonaws aws-java-sdk-directconnect 1.9.40
com.amazonaws aws-java-sdk-directory 1.9.40
com.amazonaws aws-java-sdk-dynamodb 1.9.40
com.amazonaws aws-java-sdk-ec2 1.9.40
com.amazonaws aws-java-sdk-ecs 1.9.40
com.amazonaws aws-java-sdk-efs 1.9.40
com.amazonaws aws-java-sdk-elasticache 1.9.40
com.amazonaws aws-java-sdk-elasticbeanstalk 1.9.40
com.amazonaws aws-java-sdk-elasticloadbalancing 1.9.40
com.amazonaws aws-java-sdk-elastictranscoder 1.9.40
com.amazonaws aws-java-sdk-emr 1.9.40
com.amazonaws aws-java-sdk-glacier 1.9.40
com.amazonaws aws-java-sdk-iam 1.9.40
com.amazonaws aws-java-sdk-importexport 1.9.40
com.amazonaws aws-java-sdk-kinesis 1.9.40
com.amazonaws aws-java-sdk-kms 1.9.40
com.amazonaws aws-java-sdk-lambda 1.9.40
com.amazonaws aws-java-sdk-logs 1.9.40
com.amazonaws aws-java-sdk-machinelearning 1.9.40
com.amazonaws aws-java-sdk-opsworks 1.9.40
com.amazonaws aws-java-sdk-rds 1.9.40
com.amazonaws aws-java-sdk-redshift 1.9.40
com.amazonaws aws-java-sdk-route53 1.9.40
com.amazonaws aws-java-sdk-s3 1.9.40
com.amazonaws aws-java-sdk-ses 1.9.40
com.amazonaws aws-java-sdk-simpledb 1.9.40
com.amazonaws aws-java-sdk-simpleworkflow 1.9.40
com.amazonaws aws-java-sdk-sns 1.9.40
com.amazonaws aws-java-sdk-sqs 1.9.40
com.amazonaws aws-java-sdk-ssm 1.9.40
com.amazonaws aws-java-sdk-storagegateway 1.9.40
com.amazonaws aws-java-sdk-sts 1.9.40
com.amazonaws aws-java-sdk-support 1.9.40
com.amazonaws aws-java-sdk-swf-libraries 1.9.40
com.amazonaws aws-java-sdk-workspaces 1.9.40
com.chuusai shapeless_2.11 2.0.0
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.11 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.11 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.4.5
com.fasterxml.jackson.core jackson-core 2.4.5
com.fasterxml.jackson.core jackson-databind 2.4.5
com.fasterxml.jackson.datatype jackson-datatype-joda 2.4.5
com.fasterxml.jackson.module jackson-module-paranamer 2.4.5
com.fasterxml.jackson.module jackson-module-scala_2.11 2.4.5
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.11 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.11 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.11 6.23.0
com.twitter util-core_2.11 6.23.0
com.twitter util-jvm_2.11 6.23.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging-api_2.11 2.1.2
com.typesafe.scala-logging scala-logging-slf4j_2.11 2.1.2
com.univocity univocity-parsers 2.2.1
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.8.0.Final
io.netty netty-all 4.0.42.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.java.dev.jets3t jets3t 0.7.1
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.py4j py4j 0.10.4
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-mapred 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.1
org.apache.parquet parquet-common 1.8.1
org.apache.parquet parquet-encoding 1.8.1
org.apache.parquet parquet-format 2.3.0-incubating
org.apache.parquet parquet-hadoop 1.8.1
org.apache.parquet parquet-jackson 1.8.1
org.apache.thrift libfb303 0.9.2
org.apache.thrift libthrift 0.9.2
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.3.v20150827
org.eclipse.jetty jetty-continuation 9.3.3.v20150827
org.eclipse.jetty jetty-http 9.3.3.v20150827
org.eclipse.jetty jetty-io 9.3.3.v20150827
org.eclipse.jetty jetty-jndi 9.3.3.v20150827
org.eclipse.jetty jetty-plus 9.3.3.v20150827
org.eclipse.jetty jetty-proxy 9.3.3.v20150827
org.eclipse.jetty jetty-security 9.3.3.v20150827
org.eclipse.jetty jetty-server 9.3.3.v20150827
org.eclipse.jetty jetty-servlet 9.3.3.v20150827
org.eclipse.jetty jetty-servlets 9.3.3.v20150827
org.eclipse.jetty jetty-util 9.3.3.v20150827
org.eclipse.jetty jetty-webapp 9.3.3.v20150827
org.eclipse.jetty jetty-xml 9.3.3.v20150827
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.11 3.2.11
org.json4s json4s-core_2.11 3.2.11
org.json4s json4s-jackson_2.11 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.11 2.11.8
org.scala-lang scala-library_2.11 2.11.8
org.scala-lang scala-reflect_2.11 2.11.8
org.scala-lang scalap_2.11 2.11.8
org.scala-lang.modules scala-parser-combinators_2.11 1.0.2
org.scala-lang.modules scala-xml_2.11 1.0.2
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.11 1.12.5
org.scalanlp breeze-macros_2.11 0.12
org.scalanlp breeze_2.11 0.12
org.scalatest scalatest_2.11 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.11 0.7.4
org.spire-math spire_2.11 0.7.4
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
stax stax-api 1.0.1
xerces xercesImpl 2.9.1
xml-apis xml-apis 1.3.04
xmlenc xmlenc 0.52