Databricks Runtime 3.3

Databricks released this image in early October, 2017

Important

This release was deprecated on July 30, 2018. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks Runtime Versioning and Support Lifecycle.

The following release notes provide information about the Databricks Runtime 3.3 powered by Apache Spark.

Changes and Improvements

  • DBIO Parquet Caching: Up to now, the speed at which the files from cloud storage systems or remote sources could be read was limited by the network connection latency and bandwidth. The DBIO caching can remove this limitation and significantly improve scan performance by automatically creating copies of remote data in the nodes’ local storage using fast intermediate data format. Successive read operations are then satisfied with local data, with no need to make remote network connections. To learn more see Optimizing Performance with Caching.
  • ML Model Export API for Python (Private Preview):
    • Model Export (previously called Model Scoring) allows users to export MLlib models and Pipelines to execute outside of Apache Spark in custom deployment environments. Please contact your account representative if you are interested in participating in the Alpha release of this feature.
    • This release adds a Python API for exporting models and Pipelines (dbmlModelExport.ModelExport).
  • TensorBoard integration now supports TensorFlow versions 1.1 to 1.3. See the Databricks Guide section on TensorBoard for details on using it in Databricks.
  • More than 100x performance improvement of SparkR::createDataFrame(). No code change is required, and all your existing notebooks and jobs that use this API will benefit from this improvement.
  • Fixed Desc Extended/Formatted when the table owner is null.
  • Fixed performance issue caused by partition listing during insertion of Hive SerDe tables
  • Upgraded Python library tornado from 4.5.1 to 4.5.2.

Apache Spark

The Databricks Runtime 3.3 includes Apache Spark 2.2.0. In addition to Databricks Runtime 3.2, Databricks Runtime 3.3 also includes the following extra bug fixes and improvements made to Spark:

  • [SPARK-22003][SQL] support array column in vectorized reader with UDF
  • [SPARK-21845][SQL] Make codegen fallback of expressions configurable
  • [SPARK-17642][SQL] Support DESC EXTENDED/FORMATTED table column commands
  • [SPARK-4131] Support “Writing data into the filesystem from queries”
  • [SPARK-22017][SS] Take minimum of all watermark execs in StreamExecution
  • [SPARK-22018][SQL] Preserve top-level alias metadata when collapsing projects
  • [SPARK-21979][SQL] Improve QueryPlanConstraints framework
  • [SPARK-18608][ML] Fix double caching
  • [SPARK-20098][PYSPARK] dataType’s typeName fix
  • [SPARK-21954][SQL] JacksonUtils should verify MapType’s value type instead of key type
  • [SPARK-21915][ML][PYSPARK] Model 1 and Model 2 ParamMaps Missing
  • [SPARK-21890] Credentials not being passed to add the tokens
  • [SPARK-21901][SS] Define toString for StateOperatorProgress
  • [SPARK-21925] Update trigger interval documentation in docs with behavior change in Spark 2.2
  • [SPARK-21418][SQL] NoSuchElementException: None.get in DataSourceScanExec with sun.io.serialization.extendedDebugInfo=true
  • [SPARK-21765] Check that optimization doesn’t affect isStreaming bit.
  • [SPARK-21891][SQL] Add TBLPROPERTIES to DDL statement: CREATE TABLE USING
  • [SPARK-21884][SPARK-21477][SQL] Mark LocalTableScanExec’s input data transient
  • [SPARK-21834] Incorrect executor request in case of dynamic allocation
  • [SPARK-21254][WEBUI] History UI performance fixes
  • [SPARK-21798] No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server
  • [SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSummarizer.variance generate negative result
  • [SPARK-21681][ML] fix bug of MLOR do not work correctly when featureStd contains zero
  • [SPARK-21826][SQL] outer broadcast hash join should not throw NPE
  • [SPARK-21805][SPARKR] Disable R vignettes code on Windows
  • [SPARK-21848][SQL] Add trait UserDefinedExpression to identify user-defined functions
  • [SPARK-21350][SPARK-20586][SQL] Add deterministic to ScalaUDF
  • [SPARK-21807][SQL] Override ++ operation in ExpressionSet to reduce clone time
  • [SPARK-21765] Set isStreaming in KinesisSource.
  • [SPARK-21765] Set isStreaming bit for LogicalPlan leaves from streaming sources
  • [SPARK-21788][SS] Handle more exceptions when stopping a streaming query
  • [SPARK-21617][SQL] Store correct table metadata when altering schema in Hive metastore.
  • [SPARK-21739][SQL] Cast expression should initialize timezoneId when it is called statically to convert something into TimestampType
  • [SPARK-18464][SQL] support old table which doesn’t store schema in table properties
  • [SPARK-21656][CORE] spark dynamic allocation should not idle timeout executors when tasks still to run
  • [SPARK-21723][ML] Fix writing LibSVM (key not found: numFeatures)
  • [SPARK-21721][SQL] Clear FileSystem deleteOnExit cache when paths are successfully removed
  • [SPARK-21696][SS] Fix a potential issue that may generate partial snapshot files
  • [SPARK-21563][CORE] Fix race condition when serializing TaskDescriptions and adding jars
  • [SPARK-18394][SQL] Make an AttributeSet.toSeq output order consistent
  • [SPARK-21743][SQL] top-most limit should not cause memory leak
  • [SPARK-19451] rangeBetween method should accept Long value as boundary

Known Issues

  • Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

Maintenance Updates

Maintenance updates made to Databricks Runtime 3.3 since its initial release include:

  • May 31, 2018
    • Fixed a bug affecting Spark SQL execution engine.
  • Apr 24, 2018
    • Fixed a bug affecting inserting partitioned Hive table with overwrite when spark.databricks.io.hive.fastwriter.enabled is false.
  • Mar 12, 2018
    • Fixed an issue caused by a race condition that could, in rare circumstances, lead to loss of some output files.
  • Jan 29, 2018
    • Fixed an issue affecting UDFs in Scala.
  • Oct 11, 2017
    • Improved shuffle stability.

System Environment

  • Operating System: Ubuntu 16.04.3 LTS
  • Java: 1.8.0_131
  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
  • Python: 2.7.12 (or 3.5.2 if Python 3 support is enabled)
  • R: R version 3.2.3 (2015-12-10)
  • For GPU clusters, the following NVIDIA GPU libraries are installed:
    • CUDA 8.0
    • CUDNN 6.0

Pre-installed Python Libraries

Library Version Library Version Library Version
ansi2html 1.1.1 argparse 1.2.1 backports-abc 0.5
boto 2.42.0 boto3 1.4.1 botocore 1.4.70
brewer2mpl 1.4.1 certifi 2016.2.28 cffi 1.7.0
chardet 2.3.0 colorama 0.3.7 configobj 5.0.6
cryptography 1.5 cycler 0.10.0 Cython 0.24.1
decorator 4.0.10 docutils 0.14 enum34 1.1.6
et-xmlfile 1.0.1 freetype-py 1.0.2 funcsigs 1.0.2
fusepy 2.0.4 futures 3.1.1 ggplot 0.6.8
html5lib 0.999 idna 2.1 ipaddress 1.0.16
ipython 2.2.0 ipython-genutils 0.1.0 jdcal 1.2
Jinja2 2.8 jmespath 0.9.0 llvmlite 0.13.0
lxml 3.6.4 MarkupSafe 0.23 matplotlib 1.5.3
mpld3 0.2 msgpack-python 0.4.7 ndg-httpsclient 0.3.3
numba 0.28.1 numpy 1.11.1 openpyxl 2.3.2
pandas 0.18.1 pathlib2 2.1.0 patsy 0.4.1
pexpect 4.0.1 pickleshare 0.7.4 Pillow 3.3.1
pip 9.0.1 ply 3.9 prompt-toolkit 1.0.7
psycopg2 2.6.2 ptyprocess 0.5.1 py4j 0.10.3
pyarrow 0.4.1 pyasn1 0.1.9 pycparser 2.14
Pygments 2.1.3 PyGObject 3.20.0 pyOpenSSL 16.0.0
pyparsing 2.2.0 pypng 0.0.18 Python 2.7.12
python-dateutil 2.5.3 python-geohash 0.8.5 pytz 2016.6.1
requests 2.11.1 s3transfer 0.1.9 scikit-learn 0.18.1
scipy 0.18.1 scour 0.32 seaborn 0.7.1
setuptools 36.2.7 simplejson 3.8.2 simples3 1.0
singledispatch 3.4.0.3 six 1.10.0 statsmodels 0.6.1
tornado 4.5.2 traitlets 4.3.0 urllib3 1.19.1
virtualenv 15.0.1 wcwidth 0.1.7 wheel 0.30.0a0
wsgiref 0.1.2        

Pre-installed R Libraries

Library Version Library Version Library Version
abind 1.4-3 assertthat 0.1 base 3.2.3
BH 1.60.0-2 bitops 1.0-6 boot 1.3-17
brew 1.0-6 car 2.1-3 caret 6.0-71
chron 2.3-47 class 7.3-14 cluster 2.0.5
codetools 0.2-14 colorspace 1.2-4 compiler 3.2.3
crayon 1.3.1 curl 2.2 data.table 1.9.6
datasets 3.2.3 DBI 0.5-1 devtools 1.12.0
dichromat 2.0-0 digest 0.6.9 doMC 1.3.4
dplyr 0.5.0 foreach 1.4.3 foreign 0.8-66
gbm 2.1.1 ggplot2 2.1.0 git2r 0.15.0
glmnet 2.0-5 graphics 3.2.3 grDevices 3.2.3
grid 3.2.3 gsubfn 0.6-6 gtable 0.1.2
h2o 3.10.0.8 httr 1.2.1 hwriter 1.3.2
hwriterPlus 1.0-3 iterators 1.0.8 jsonlite 1.1
KernSmooth 2.23-15 labeling 0.3 lattice 0.20-34
lazyeval 0.2.0 littler 0.3.0 lme4 1.1-12
lubridate 1.6.0 magrittr 1.5 mapproj 1.2-4
maps 3.0.2 MASS 7.3-45 Matrix 1.2-7.1
MatrixModels 0.4-1 memoise 1.0.0 methods 3.2.3
mgcv 1.8-11 mime 0.5 minqa 1.2.4
multicore 0.2 munsell 0.4.2 mvtnorm 1.0-5
nlme 3.1-124 nloptr 1.0.4 nnet 7.3-12
openssl 0.9.4 parallel 3.2.3 pbkrtest 0.4-6
pkgKitten 0.1.3 plyr 1.8.4 praise 1.0.0
pROC 1.8 proto 0.3-10 quantreg 5.29
R.methodsS3 1.7.1 R.oo 1.20.0 R.utils 2.4.0
R6 2.2.0 randomForest 4.6-12 RColorBrewer 1.1-2
Rcpp 0.12.7 RcppEigen 0.3.2.9.0 RCurl 1.95-4.8
reshape2 1.4.2 RODBC 1.3-12 roxygen2 5.0.1
rpart 4.1-10 Rserve 1.7-3 RSQLite 1.0.0
rstudioapi 0.6 scales 0.3.0 sp 1.0-15
SparkR 2.2.0 SparseM 1.72 spatial 7.3-11
splines 3.2.3 sqldf 0.4-10 statmod 1.4.26
stats 3.2.3 stats4 3.2.3 stringi 1.0-1
stringr 1.0.0 survival 2.38-3 tcltk 3.2.3
TeachingDemos 2.10 testthat 1.0.2 tibble 1.2
tools 3.2.3 utils 3.2.3 whisker 0.3-2
withr 1.0.2        

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.7.3
com.amazonaws aws-java-sdk-autoscaling 1.11.126
com.amazonaws aws-java-sdk-cloudformation 1.11.126
com.amazonaws aws-java-sdk-cloudfront 1.11.126
com.amazonaws aws-java-sdk-cloudhsm 1.11.126
com.amazonaws aws-java-sdk-cloudsearch 1.11.126
com.amazonaws aws-java-sdk-cloudtrail 1.11.126
com.amazonaws aws-java-sdk-cloudwatch 1.11.126
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.126
com.amazonaws aws-java-sdk-codedeploy 1.11.126
com.amazonaws aws-java-sdk-cognitoidentity 1.11.126
com.amazonaws aws-java-sdk-cognitosync 1.11.126
com.amazonaws aws-java-sdk-config 1.11.126
com.amazonaws aws-java-sdk-core 1.11.126
com.amazonaws aws-java-sdk-datapipeline 1.11.126
com.amazonaws aws-java-sdk-directconnect 1.11.126
com.amazonaws aws-java-sdk-directory 1.11.126
com.amazonaws aws-java-sdk-dynamodb 1.11.126
com.amazonaws aws-java-sdk-ec2 1.11.126
com.amazonaws aws-java-sdk-ecs 1.11.126
com.amazonaws aws-java-sdk-efs 1.11.126
com.amazonaws aws-java-sdk-elasticache 1.11.126
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.126
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.126
com.amazonaws aws-java-sdk-elastictranscoder 1.11.126
com.amazonaws aws-java-sdk-emr 1.11.126
com.amazonaws aws-java-sdk-glacier 1.11.126
com.amazonaws aws-java-sdk-iam 1.11.126
com.amazonaws aws-java-sdk-importexport 1.11.126
com.amazonaws aws-java-sdk-kinesis 1.11.126
com.amazonaws aws-java-sdk-kms 1.11.126
com.amazonaws aws-java-sdk-lambda 1.11.126
com.amazonaws aws-java-sdk-logs 1.11.126
com.amazonaws aws-java-sdk-machinelearning 1.11.126
com.amazonaws aws-java-sdk-opsworks 1.11.126
com.amazonaws aws-java-sdk-rds 1.11.126
com.amazonaws aws-java-sdk-redshift 1.11.126
com.amazonaws aws-java-sdk-route53 1.11.126
com.amazonaws aws-java-sdk-s3 1.11.126
com.amazonaws aws-java-sdk-ses 1.11.126
com.amazonaws aws-java-sdk-simpledb 1.11.126
com.amazonaws aws-java-sdk-simpleworkflow 1.11.126
com.amazonaws aws-java-sdk-sns 1.11.126
com.amazonaws aws-java-sdk-sqs 1.11.126
com.amazonaws aws-java-sdk-ssm 1.11.126
com.amazonaws aws-java-sdk-storagegateway 1.11.126
com.amazonaws aws-java-sdk-sts 1.11.126
com.amazonaws aws-java-sdk-support 1.11.126
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.126
com.amazonaws jmespath-java 1.11.126
com.carrotsearch hppc 0.7.1
com.chuusai shapeless_2.10 2.3.2
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.10 0.2.2-db1-spark2.2
com.databricks dbml-local_2.10-tests 0.2.2-db1-spark2.2
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.10 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.10 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.6.7
com.fasterxml.jackson.core jackson-core 2.6.7
com.fasterxml.jackson.core jackson-databind 2.6.7.1
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.6.7
com.fasterxml.jackson.datatype jackson-datatype-joda 2.6.7
com.fasterxml.jackson.module jackson-module-paranamer 2.6.7
com.fasterxml.jackson.module jackson-module-scala_2.10 2.6.7.1
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jamesmurty.utils java-xmlbuilder 1.0
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.microsoft.azure azure-data-lake-store-sdk 2.0.11
com.microsoft.sqlserver mssql-jdbc 6.1.0.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.10 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.10 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.10 6.23.0
com.twitter util-core_2.10 6.23.0
com.twitter util-jvm_2.10 6.23.0
com.typesafe config 1.2.1
com.typesafe scalalogging-slf4j_2.10 1.1.0
com.univocity univocity-parsers 2.2.1
com.vlkan flatbuffers 1.2.0-3f79e055
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.9.9.Final
io.netty netty-all 4.0.43.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mx4j mx4j 3.0.2
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.iharder base64 2.3.8
net.java.dev.jets3t jets3t 0.9.3
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.4.0
org.apache.arrow arrow-memory 0.4.0
org.apache.arrow arrow-vector 0.4.0
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.2
org.apache.parquet parquet-common 1.8.2
org.apache.parquet parquet-encoding 1.8.2
org.apache.parquet parquet-format 2.3.1
org.apache.parquet parquet-hadoop 1.8.2
org.apache.parquet parquet-jackson 1.8.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.9.3
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.bouncycastle bcprov-jdk15on 1.51
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.11.v20160721
org.eclipse.jetty jetty-continuation 9.3.11.v20160721
org.eclipse.jetty jetty-http 9.3.11.v20160721
org.eclipse.jetty jetty-io 9.3.11.v20160721
org.eclipse.jetty jetty-jndi 9.3.11.v20160721
org.eclipse.jetty jetty-plus 9.3.11.v20160721
org.eclipse.jetty jetty-proxy 9.3.11.v20160721
org.eclipse.jetty jetty-security 9.3.11.v20160721
org.eclipse.jetty jetty-server 9.3.11.v20160721
org.eclipse.jetty jetty-servlet 9.3.11.v20160721
org.eclipse.jetty jetty-servlets 9.3.11.v20160721
org.eclipse.jetty jetty-util 9.3.11.v20160721
org.eclipse.jetty jetty-webapp 9.3.11.v20160721
org.eclipse.jetty jetty-xml 9.3.11.v20160721
org.fusesource.jansi jansi 1.4
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.10 3.2.11
org.json4s json4s-core_2.10 3.2.11
org.json4s json4s-jackson_2.10 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rocksdb rocksdbjni 5.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang jline 2.10.6
org.scala-lang scala-compiler_2.10 2.10.6
org.scala-lang scala-library_2.10 2.10.6
org.scala-lang scala-reflect_2.10 2.10.6
org.scala-lang scalap_2.10 2.10.6
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.10 1.12.5
org.scalamacros quasiquotes_2.10 2.0.0
org.scalanlp breeze-macros_2.10 0.13.2
org.scalanlp breeze_2.10 0.13.2
org.scalatest scalatest_2.10 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.10 0.13.0
org.spire-math spire_2.10 0.13.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.typelevel machinist_2.10 0.6.1
org.typelevel macro-compat_2.10 1.1.1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.7.3
com.amazonaws aws-java-sdk-autoscaling 1.11.126
com.amazonaws aws-java-sdk-cloudformation 1.11.126
com.amazonaws aws-java-sdk-cloudfront 1.11.126
com.amazonaws aws-java-sdk-cloudhsm 1.11.126
com.amazonaws aws-java-sdk-cloudsearch 1.11.126
com.amazonaws aws-java-sdk-cloudtrail 1.11.126
com.amazonaws aws-java-sdk-cloudwatch 1.11.126
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.126
com.amazonaws aws-java-sdk-codedeploy 1.11.126
com.amazonaws aws-java-sdk-cognitoidentity 1.11.126
com.amazonaws aws-java-sdk-cognitosync 1.11.126
com.amazonaws aws-java-sdk-config 1.11.126
com.amazonaws aws-java-sdk-core 1.11.126
com.amazonaws aws-java-sdk-datapipeline 1.11.126
com.amazonaws aws-java-sdk-directconnect 1.11.126
com.amazonaws aws-java-sdk-directory 1.11.126
com.amazonaws aws-java-sdk-dynamodb 1.11.126
com.amazonaws aws-java-sdk-ec2 1.11.126
com.amazonaws aws-java-sdk-ecs 1.11.126
com.amazonaws aws-java-sdk-efs 1.11.126
com.amazonaws aws-java-sdk-elasticache 1.11.126
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.126
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.126
com.amazonaws aws-java-sdk-elastictranscoder 1.11.126
com.amazonaws aws-java-sdk-emr 1.11.126
com.amazonaws aws-java-sdk-glacier 1.11.126
com.amazonaws aws-java-sdk-iam 1.11.126
com.amazonaws aws-java-sdk-importexport 1.11.126
com.amazonaws aws-java-sdk-kinesis 1.11.126
com.amazonaws aws-java-sdk-kms 1.11.126
com.amazonaws aws-java-sdk-lambda 1.11.126
com.amazonaws aws-java-sdk-logs 1.11.126
com.amazonaws aws-java-sdk-machinelearning 1.11.126
com.amazonaws aws-java-sdk-opsworks 1.11.126
com.amazonaws aws-java-sdk-rds 1.11.126
com.amazonaws aws-java-sdk-redshift 1.11.126
com.amazonaws aws-java-sdk-route53 1.11.126
com.amazonaws aws-java-sdk-s3 1.11.126
com.amazonaws aws-java-sdk-ses 1.11.126
com.amazonaws aws-java-sdk-simpledb 1.11.126
com.amazonaws aws-java-sdk-simpleworkflow 1.11.126
com.amazonaws aws-java-sdk-sns 1.11.126
com.amazonaws aws-java-sdk-sqs 1.11.126
com.amazonaws aws-java-sdk-ssm 1.11.126
com.amazonaws aws-java-sdk-storagegateway 1.11.126
com.amazonaws aws-java-sdk-sts 1.11.126
com.amazonaws aws-java-sdk-support 1.11.126
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.126
com.amazonaws jmespath-java 1.11.126
com.carrotsearch hppc 0.7.1
com.chuusai shapeless_2.11 2.3.2
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.11 0.2.2-db1-spark2.2
com.databricks dbml-local_2.11-tests 0.2.2-db1-spark2.2
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.11 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.11 0.4.15-9
com.esotericsoftware kryo-shaded 3.0.3
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.6.7
com.fasterxml.jackson.core jackson-core 2.6.7
com.fasterxml.jackson.core jackson-databind 2.6.7.1
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.6.7
com.fasterxml.jackson.datatype jackson-datatype-joda 2.6.7
com.fasterxml.jackson.module jackson-module-paranamer 2.6.7
com.fasterxml.jackson.module jackson-module-scala_2.11 2.6.7.1
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jamesmurty.utils java-xmlbuilder 1.0
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.mchange c3p0 0.9.5.1
com.mchange mchange-commons-java 0.2.10
com.microsoft.azure azure-data-lake-store-sdk 2.0.11
com.microsoft.sqlserver mssql-jdbc 6.1.0.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.6
com.trueaccord.lenses lenses_2.11 0.3
com.twitter chill-java 0.8.0
com.twitter chill_2.11 0.8.0
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.11 6.23.0
com.twitter util-core_2.11 6.23.0
com.twitter util-jvm_2.11 6.23.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging-api_2.11 2.1.2
com.typesafe.scala-logging scala-logging-slf4j_2.11 2.1.2
com.univocity univocity-parsers 2.2.1
com.vlkan flatbuffers 1.2.0-3f79e055
com.zaxxer HikariCP 2.4.1
commons-beanutils commons-beanutils 1.7.0
commons-beanutils commons-beanutils-core 1.8.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 2.2
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.dropwizard.metrics metrics-core 3.1.2
io.dropwizard.metrics metrics-ganglia 3.1.2
io.dropwizard.metrics metrics-graphite 3.1.2
io.dropwizard.metrics metrics-healthchecks 3.1.2
io.dropwizard.metrics metrics-jetty9 3.1.2
io.dropwizard.metrics metrics-json 3.1.2
io.dropwizard.metrics metrics-jvm 3.1.2
io.dropwizard.metrics metrics-log4j 3.1.2
io.dropwizard.metrics metrics-servlets 3.1.2
io.netty netty 3.9.9.Final
io.netty netty-all 4.0.43.Final
io.prometheus simpleclient 0.0.16
io.prometheus simpleclient_common 0.0.16
io.prometheus simpleclient_dropwizard 0.0.16
io.prometheus simpleclient_servlet 0.0.16
io.prometheus.jmx collector 0.7
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.11
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
mx4j mx4j 3.0.2
mysql mysql-connector-java 5.1.27
net.hydromatic eigenbase-properties 1.1.5
net.iharder base64 2.3.8
net.java.dev.jets3t jets3t 0.9.3
net.jpountz.lz4 lz4 1.3.0
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.5.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.4.0
org.apache.arrow arrow-memory 0.4.0
org.apache.arrow arrow-vector 0.4.0
org.apache.avro avro 1.7.7
org.apache.avro avro-ipc 1.7.7
org.apache.avro avro-ipc-tests 1.7.7
org.apache.avro avro-mapred-hadoop2 1.7.7
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.4.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.6.0
org.apache.curator curator-framework 2.6.0
org.apache.curator curator-recipes 2.6.0
org.apache.derby derby 10.10.2.0
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.2
org.apache.httpcomponents httpcore 4.4.4
org.apache.ivy ivy 2.4.0
org.apache.parquet parquet-column 1.8.2
org.apache.parquet parquet-common 1.8.2
org.apache.parquet parquet-encoding 1.8.2
org.apache.parquet parquet-format 2.3.1
org.apache.parquet parquet-hadoop 1.8.2
org.apache.parquet parquet-jackson 1.8.2
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.9.3
org.apache.xbean xbean-asm5-shaded 4.4
org.apache.zookeeper zookeeper 3.4.6
org.bouncycastle bcprov-jdk15on 1.51
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.0
org.codehaus.janino janino 3.0.0
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.11.v20160721
org.eclipse.jetty jetty-continuation 9.3.11.v20160721
org.eclipse.jetty jetty-http 9.3.11.v20160721
org.eclipse.jetty jetty-io 9.3.11.v20160721
org.eclipse.jetty jetty-jndi 9.3.11.v20160721
org.eclipse.jetty jetty-plus 9.3.11.v20160721
org.eclipse.jetty jetty-proxy 9.3.11.v20160721
org.eclipse.jetty jetty-security 9.3.11.v20160721
org.eclipse.jetty jetty-server 9.3.11.v20160721
org.eclipse.jetty jetty-servlet 9.3.11.v20160721
org.eclipse.jetty jetty-servlets 9.3.11.v20160721
org.eclipse.jetty jetty-util 9.3.11.v20160721
org.eclipse.jetty jetty-webapp 9.3.11.v20160721
org.eclipse.jetty jetty-xml 9.3.11.v20160721
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.jpmml pmml-model 1.2.15
org.jpmml pmml-schema 1.2.15
org.json4s json4s-ast_2.11 3.2.11
org.json4s json4s-core_2.11 3.2.11
org.json4s json4s-jackson_2.11 3.2.11
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.1
org.postgresql postgresql 9.4-1204-jdbc41
org.roaringbitmap RoaringBitmap 0.5.11
org.rocksdb rocksdbjni 5.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.11 2.11.8
org.scala-lang scala-library_2.11 2.11.8
org.scala-lang scala-reflect_2.11 2.11.8
org.scala-lang scalap_2.11 2.11.8
org.scala-lang.modules scala-parser-combinators_2.11 1.0.2
org.scala-lang.modules scala-xml_2.11 1.0.2
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.11 1.12.5
org.scalanlp breeze-macros_2.11 0.13.2
org.scalanlp breeze_2.11 0.13.2
org.scalatest scalatest_2.11 2.2.6
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-exec 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.11 0.13.0
org.spire-math spire_2.11 0.13.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.0
org.typelevel machinist_2.11 0.6.1
org.typelevel macro-compat_2.11 1.1.1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.2.6
org.yaml snakeyaml 1.16
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52