Databricks Runtime 5.5 LTS

Databricks released this image in July 2019. It was declared Long Term Support (LTS) in August 2019.

The following release notes provide information about Databricks Runtime 5.5, powered by Apache Spark.

New Features

Delta Lake on Databricks Auto Optimize

Today, when you write data into cloud storage, you need to compact files for optimal I/O performance. You have to worry about the right file size, how often to compact the files, how big a cluster to use, and so on. To address this class of problems, we are excited to announce the general availability of Auto Optimize with Delta Lake on Databricks. During every write to Delta tables, we automatically figure out the right file sizes and compact files so that you don’t have to worry about optimizing the storage layout. During writes, if the auto-optimize option is true, Databricks automatically determines if an optimize is required and optimizes the small files. For details, see Auto Optimize.

Delta Lake on Databricks improved min, max, and count aggregation query performance

The performance of min, max, and count aggregation queries for Delta Lake on Databricks has been significantly improved by reducing the amount of data that’s read. These queries are now run using statistics and partition values in the metadata, rather than scanning the data.

Presto and Athena support for Delta tables on AWS S3 (Public Preview)

You can now query Delta tables from external tools such as Presto and Athena. Presto and Athena support reading from external tables when the list of data files to process is read from a manifest file, which is a text file containing the list of data files to read for querying a table. When an external table is defined in the Hive metastore using manifest files, Presto and Athena use the list of files in the manifest rather than finding the files by directory listing. See Presto and Athena Compatibility Support for Delta Lake for details.

Glue Catalog as Databricks metastore

Glue Catalog support is generally available. This feature lets you configure Databricks Runtime to use the AWS Glue Data Catalog as its metastore, which can serve as a drop-in replacement for an external Hive metastore. It also enables multiple Databricks workspaces to share the same metastore. For more information and limitations, see Using AWS Glue Data Catalog as the Metastore for Databricks Runtime.

Preview

Credential passthrough for Glue Catalog is in Private Preview. Contact support if you’d like to try this out.

Faster model inference pipelines with improved binary file data source and scalar iterator pandas UDF (Public Preview)

Machine learning tasks, especially in the image and video domain, often have to operate on a large number of files. In Databricks Runtime 5.4, we already made available the binary file data source to help ETL arbitrary files such as images, into Spark tables. In Databricks Runtime 5.5, we have added an option, recursiveFileLookup, to load files recursively from nested input directories. See Binary Files.

The binary file data source enables you to run model inference tasks in parallel from Spark tables using a scalar pandas UDF. However, you might have to initialize the model for every record batch, which introduces overhead. In Databricks Runtime 5.5, we are backporting a new pandas UDF type called “scalar iterator” from Apache Spark master. With it you can initialize the model only once and apply the model to many input batches, which can result in a 2-3x speedup for models like ResNet50. See Scalar iterator UDFs.

Secrets API in R notebooks

The Secrets API lets you inject secrets into notebooks without hardcoding them. This API is now available in R notebooks in addition to existing support for Python and Scala notebooks. You can use the dbutils.secrets.get function to obtain secrets. Secrets are redacted before printing to a notebook cell.

Improvements

  • Support for running Delta Lake SQL operations in Python foreachBatch: We have fixed the known limitation of not being able to write to Delta tables from inside foreachBatch of a Structured Streaming query defined in Python. This is useful in a common Python streaming workloads; for example, Writing streaming aggregates in update mode using MERGE and foreachBatch.
  • Performance of Delta tables stored in Azure Data Lake Gen2: The check for the latest version of a Delta table on ADLS Gen2 now only checks the end of the transaction log, rather than listing all available versions. This optimization makes UPDATE a constant time operation, significantly improving the latency.
  • Scalability of Optimize ZORDER BY: Z-Ordering on very large Delta tables now uses smaller units of work that are governed by advanced admission control. This feature improves the stability of this operation without sacrificing cluster utilization.
  • Improved performance of DML commands on tables with large number of columns: We now perform better column pruning when scanning for matching data in UPDATE, DELETE, and MERGE commands.
  • Support for VNet + Service Endpoints setup in Spark - SQL DW connector: We white-listed ADL Gen2 paths as temp data locations (.option("tempDir", "abfss://...")) and added a new option called useAzureMSI, to be used instead of forward_spark_azure_storage_credentials in case your SQL DW is configured to authenticate via Managed Identities to your V2 storage account.
  • Automatic invalidation for Delta cache: The Delta cache now automatically detects files that have been modified or overwritten after being cached. Any stale entries are automatically invalidated and evicted from the cache. See Optimizing Performance with Caching.
  • Upgraded Python library wheel from 0.33.3 to 0.33.4.
  • Upgraded R library nlme from 3.1-139 to 3.1-140.

Bug fixes

  • Fixed cancellation for R commands that do not run Spark jobs. Previously, R commands that do not run Spark jobs could be cancelled, but the notebooks’ state would be lost; commands can now be cancelled without losing notebook state.
  • Dropping or moving a managed table now invalidates the cached Delta Lake log.
  • Fixed a bug that writing Delta Lake checkpoint may fail due to FileAlreadyExistsException.
  • Scala REPL now sets the proper -target:jvm-1.8 flag to support calling Java methods that use Java 8 features.
  • Fixed a bug that the S3-SQS/ABS-AQS connector may hit a corrupt Delta table error when a file gets removed and re-added.

Apache Spark

Databricks Runtime 5.5 includes Apache Spark 2.4.3. This release includes all Spark fixes and improvements included in Databricks Runtime 5.4, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-24695] Allow UDFs to return CalendarInterval
  • [SPARK-28056] add docstring/doctest for SCALAR_ITER pandas UDF
  • [SPARK-28185] Closes the generator when Python UDFs stop early
  • [SPARK-24703] Support for interval multiplication
  • [SPARK-27018][CORE] Fix incorrect removal of checkpointed file in PeriodicCheckpointer
  • [SPARK-28127][SQL] Micro optimization on TreeNode’s mapChildren method
  • [SPARK-26038] Decimal toScalaBigInt/toJavaBigInteger for decimals not fitting in long
  • [SPARK-26555][SQL] make ScalaReflection subtype checking thread safe
  • [SPARK-28081][ML] Handle large vocab counts in word2vec
  • [SPARK-21882][CORE] OutputMetrics doesn’t count written bytes correctly in the saveAsHadoopDataset function
  • [SPARK-28030] convert filePath to URI in binary file data source
  • [SPARK-27803][SQL][PYTHON] Fix column pruning for Python UDF
  • [SPARK-27917][SQL] canonical form of CaseWhen object is incorrect
  • [SPARK-27798][SQL] from_avro shouldn’t produces same value when converted to local relation
  • [SPARK-27873][SQL] columnNameOfCorruptRecord should not be checked with column names in CSV header when disabling enforceSchema
  • [SPARK-27907][SQL] HiveUDAF should return NULL in case of 0 rows
  • [SPARK-27699][SQL] Partially push down disjunctive predicated in Parquet/ORC
  • [SPARK-27868][CORE] Better default value and documentation for socket server backlog.
  • [SPARK-27869][CORE] Redact sensitive information in System Properties from UI
  • [SPARK-27863][SQL][BACKPORT-2.4] Metadata files and temporary files should not be counted as data files
  • [SPARK-27657][ML] Fix the log format of ml.util.Instrumentation.logFai…
  • [SPARK-27858][SQL] Fix for Avro deserialization on union types with multiple non-null types
  • [SPARK-27711][CORE] Unset InputFileBlockHolder at the end of tasks
  • [SPARK-27351][SQL] Wrong outputRows estimation after AggregateEstimation with only null value column
  • [SPARK-27539][SQL] Fix inaccurate aggregate outputRows estimation with column containing null values”
  • [SPARK-27800][SQL] Fix wrong answer on BitwiseXor test cases
  • [SPARK-27639][SQL] InMemoryTableScan shows the table name on UI if possible
  • [SPARK-27726][CORE] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads
  • [SPARK-27771][SQL] Add SQL description for grouping functions (cube, rollup, grouping and grouping_id)
  • [SPARK-27735][SS] Parsing interval string should be case-insensitive in SS
  • [SPARK-26856][PYSPARK] Fix UT failure due to wrong patterns for Kinesis assembly
  • [SPARK-26856][PYSPARK] Python support for from_avro and to_avro APIs
  • [SPARK-26870][SQL] Move to_avro/from_avro into functions object due to Java compatibility
  • [SPARK-26812][SQL] Report correct nullability for complex datatypes in Union
  • [SPARK-27671][SQL] Fix error when casting from a nested null in a struct
  • [SPARK-27673][SQL] Add since info to random, regex, null expressions
  • [SPARK-27672][SQL] Add since info to string expressions
  • [SPARK-25139][SPARK-18406][CORE] Avoid NonFatals to kill the Executor in PythonRunner
  • [SPARK-27624][CORE] Fix CalenderInterval to show an empty interval correctly
  • [SPARK-27577][MLLIB] Correct thresholds downsampled in BinaryClassificationMetrics
  • [SPARK-27621][ML] Linear Regression - validate training related params such as loss only during fitting phase
  • [SPARK-26048][SPARK-24530] Cherrypick all the missing commits to 2.4 release script
  • [SPARK-24935][SQL] support INIT -> UPDATE -> MERGE -> FINISH in Hive UDAF adapter

System environment

  • Operating System: Ubuntu 16.04.6 LTS
  • Java: 1.8.0_212
  • Scala: 2.11.12
  • Python: 2.7.12 for Python 2 clusters and 3.5.2 for Python 3 clusters. For details, see Python version.
  • R: R version 3.6.0 (2019-04-26)
  • GPU clusters: The following NVIDIA GPU libraries are installed:
    • Tesla driver 375.66
    • CUDA 9.0
    • cuDNN 7.0

Note

Although Scala 2.12 is supported in Apache Spark 2.4, it is not supported in Databricks Runtime 5.5.

Installed Python libraries

Library Version Library Version Library Version
ansi2html 1.1.1 argparse 1.2.1 backports-abc 0.5
boto 2.42.0 boto3 1.4.1 botocore 1.4.70
brewer2mpl 1.4.1 certifi 2016.2.28 cffi 1.7.0
chardet 2.3.0 colorama 0.3.7 configobj 5.0.6
cryptography 1.5 cycler 0.10.0 Cython 0.24.1
decorator 4.0.10 docutils 0.14 enum34 1.1.6
et-xmlfile 1.0.1 freetype-py 1.0.2 funcsigs 1.0.2
fusepy 2.0.4 futures 3.2.0 ggplot 0.6.8
html5lib 0.999 idna 2.1 ipaddress 1.0.16
ipython 2.2.0 ipython-genutils 0.1.0 jdcal 1.2
Jinja2 2.8 jmespath 0.9.0 llvmlite 0.13.0
lxml 3.6.4 MarkupSafe 0.23 matplotlib 1.5.3
mpld3 0.2 msgpack-python 0.4.7 ndg-httpsclient 0.3.3
numba 0.28.1 numpy 1.11.1 openpyxl 2.3.2
pandas 0.19.2 pathlib2 2.1.0 patsy 0.4.1
pexpect 4.0.1 pickleshare 0.7.4 Pillow 3.3.1
pip 19.1.1 ply 3.9 prompt-toolkit 1.0.7
psycopg2 2.6.2 ptyprocess 0.5.1 py4j 0.10.3
pyarrow 0.8.0 pyasn1 0.1.9 pycparser 2.14
Pygments 2.1.3 PyGObject 3.20.0 pyOpenSSL 16.0.0
pyparsing 2.2.0 pypng 0.0.18 Python 2.7.12
python-dateutil 2.5.3 python-geohash 0.8.5 pytz 2016.6.1
requests 2.11.1 s3transfer 0.1.9 scikit-learn 0.18.1
scipy 0.18.1 scour 0.32 seaborn 0.7.1
setuptools 41.0.1 simplejson 3.8.2 simples3 1.0
singledispatch 3.4.0.3 six 1.10.0 statsmodels 0.6.1
tornado 5.1.1 traitlets 4.3.0 urllib3 1.19.1
virtualenv 16.1.0 wcwidth 0.1.7 wheel 0.33.4
wsgiref 0.1.2        

Installed R libraries

Library Version Library Version Library Version
abind 1.4-5 askpass 1.1 assertthat 0.2.1
backports 1.1.3 base 3.6.0 base64enc 0.1-3
BH 1.69.0-1 bit 1.1-14 bit64 0.9-7
bitops 1.0-6 blob 1.1.1 boot 1.3-20
brew 1.0-6 callr 3.2.0 car 3.0-2
carData 3.0-2 caret 6.0-82 cellranger 1.1.0
chron 2.3-53 class 7.3-15 cli 1.1.0
clipr 0.5.0 clisymbols 1.2.0 cluster 2.0.8
codetools 0.2-16 colorspace 1.4-1 commonmark 1.7
compiler 3.6.0 config 0.3 crayon 1.3.4
curl 3.3 data.table 1.12.0 datasets 3.6.0
DBI 1.0.0 dbplyr 1.3.0 desc 1.2.0
devtools 2.0.1 digest 0.6.18 doMC 1.3.5
dplyr 0.8.0.1 ellipsis 0.1.0 fansi 0.4.0
forcats 0.4.0 foreach 1.4.4 foreign 0.8-71
forge 0.2.0 fs 1.2.7 gbm 2.1.5
generics 0.0.2 ggplot2 3.1.0 gh 1.0.1
git2r 0.25.2 glmnet 2.0-16 glue 1.3.1
gower 0.2.0 graphics 3.6.0 grDevices 3.6.0
grid 3.6.0 gridExtra 2.3 gsubfn 0.7
gtable 0.3.0 h2o 3.22.1.1 haven 2.1.0
hms 0.4.2 htmltools 0.3.6 htmlwidgets 1.3
httr 1.4.0 hwriter 1.3.2 hwriterPlus 1.0-3
ini 0.3.1 ipred 0.9-8 iterators 1.0.10
jsonlite 1.6 KernSmooth 2.23-15 labeling 0.3
lattice 0.20-38 lava 1.6.5 lazyeval 0.2.2
littler 0.3.7 lme4 1.1-21 lubridate 1.7.4
magrittr 1.5 mapproj 1.2.6 maps 3.3.0
maptools 0.9-5 MASS 7.3-51.1 Matrix 1.2-17
MatrixModels 0.4-1 memoise 1.1.0 methods 3.6.0
mgcv 1.8-28 mime 0.6 minqa 1.2.4
ModelMetrics 1.2.2 munsell 0.5.0 mvtnorm 1.0-10
nlme 3.1-140 nloptr 1.2.1 nnet 7.3-12
numDeriv 2016.8-1 openssl 1.3 openxlsx 4.1.0
parallel 3.6.0 pbkrtest 0.4-7 pillar 1.3.1
pkgbuild 1.0.3 pkgconfig 2.0.2 pkgKitten 0.1.4
pkgload 1.0.2 plogr 0.2.0 plyr 1.8.4
praise 1.0.0 prettyunits 1.0.2 pROC 1.14.0
processx 3.3.0 prodlim 2018.04.18 progress 1.2.0
proto 1.0.0 ps 1.3.0 purrr 0.3.2
quantreg 5.38 R.methodsS3 1.7.1 R.oo 1.22.0
R.utils 2.8.0 r2d3 0.2.3 R6 2.4.0
randomForest 4.6-14 rappdirs 0.3.1 rcmdcheck 1.3.2
RColorBrewer 1.1-2 Rcpp 1.0.1 RcppEigen 0.3.3.5.0
RcppRoll 0.3.0 RCurl 1.95-4.12 readr 1.3.1
readxl 1.3.1 recipes 0.1.5 rematch 1.0.1
remotes 2.0.2 reshape2 1.4.3 rio 0.5.16
rlang 0.3.3 RODBC 1.3-15 roxygen2 6.1.1
rpart 4.1-15 rprojroot 1.3-2 Rserve 1.8-6
RSQLite 2.1.1 rstudioapi 0.10 scales 1.0.0
sessioninfo 1.1.1 sp 1.3-1 sparklyr 1.0.0
SparkR 2.4.4 SparseM 1.77 spatial 7.3-11
splines 3.6.0 sqldf 0.4-11 SQUAREM 2017.10-1
statmod 1.4.30 stats 3.6.0 stats4 3.6.0
stringi 1.4.3 stringr 1.4.0 survival 2.43-3
sys 3.1 tcltk 3.6.0 TeachingDemos 2.10
testthat 2.0.1 tibble 2.1.1 tidyr 0.8.3
tidyselect 0.2.5 timeDate 3043.102 tools 3.6.0
usethis 1.4.0 utf8 1.1.4 utils 3.6.0
viridisLite 0.3.0 whisker 0.3-2 withr 2.1.2
xml2 1.2.0 xopen 1.0.0 yaml 2.2.0
zip 2.0.1        

Installed Java and Scala libraries (Scala 2.11 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.8.10
com.amazonaws aws-java-sdk-autoscaling 1.11.313
com.amazonaws aws-java-sdk-cloudformation 1.11.313
com.amazonaws aws-java-sdk-cloudfront 1.11.313
com.amazonaws aws-java-sdk-cloudhsm 1.11.313
com.amazonaws aws-java-sdk-cloudsearch 1.11.313
com.amazonaws aws-java-sdk-cloudtrail 1.11.313
com.amazonaws aws-java-sdk-cloudwatch 1.11.313
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.313
com.amazonaws aws-java-sdk-codedeploy 1.11.313
com.amazonaws aws-java-sdk-cognitoidentity 1.11.313
com.amazonaws aws-java-sdk-cognitosync 1.11.313
com.amazonaws aws-java-sdk-config 1.11.313
com.amazonaws aws-java-sdk-core 1.11.313
com.amazonaws aws-java-sdk-datapipeline 1.11.313
com.amazonaws aws-java-sdk-directconnect 1.11.313
com.amazonaws aws-java-sdk-directory 1.11.313
com.amazonaws aws-java-sdk-dynamodb 1.11.313
com.amazonaws aws-java-sdk-ec2 1.11.313
com.amazonaws aws-java-sdk-ecs 1.11.313
com.amazonaws aws-java-sdk-efs 1.11.313
com.amazonaws aws-java-sdk-elasticache 1.11.313
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.313
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.313
com.amazonaws aws-java-sdk-elastictranscoder 1.11.313
com.amazonaws aws-java-sdk-emr 1.11.313
com.amazonaws aws-java-sdk-glacier 1.11.313
com.amazonaws aws-java-sdk-glue 1.11.313
com.amazonaws aws-java-sdk-iam 1.11.313
com.amazonaws aws-java-sdk-importexport 1.11.313
com.amazonaws aws-java-sdk-kinesis 1.11.313
com.amazonaws aws-java-sdk-kms 1.11.313
com.amazonaws aws-java-sdk-lambda 1.11.313
com.amazonaws aws-java-sdk-logs 1.11.313
com.amazonaws aws-java-sdk-machinelearning 1.11.313
com.amazonaws aws-java-sdk-opsworks 1.11.313
com.amazonaws aws-java-sdk-rds 1.11.313
com.amazonaws aws-java-sdk-redshift 1.11.313
com.amazonaws aws-java-sdk-route53 1.11.313
com.amazonaws aws-java-sdk-s3 1.11.313
com.amazonaws aws-java-sdk-ses 1.11.313
com.amazonaws aws-java-sdk-simpledb 1.11.313
com.amazonaws aws-java-sdk-simpleworkflow 1.11.313
com.amazonaws aws-java-sdk-sns 1.11.313
com.amazonaws aws-java-sdk-sqs 1.11.313
com.amazonaws aws-java-sdk-ssm 1.11.313
com.amazonaws aws-java-sdk-storagegateway 1.11.313
com.amazonaws aws-java-sdk-sts 1.11.313
com.amazonaws aws-java-sdk-support 1.11.313
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.313
com.amazonaws jmespath-java 1.11.313
com.carrotsearch hppc 0.7.2
com.chuusai shapeless_2.11 2.3.2
com.clearspring.analytics stream 2.7.0
com.databricks Rserve 1.8-3
com.databricks dbml-local_2.11 0.5.0-db8-spark2.4
com.databricks dbml-local_2.11-tests 0.5.0-db8-spark2.4
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.11 0.4.15-9
com.databricks.scalapb scalapb-runtime_2.11 0.4.15-9
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.0.0
com.fasterxml.jackson.core jackson-annotations 2.6.7
com.fasterxml.jackson.core jackson-core 2.6.7
com.fasterxml.jackson.core jackson-databind 2.6.7.1
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.6.7
com.fasterxml.jackson.datatype jackson-datatype-joda 2.6.7
com.fasterxml.jackson.module jackson-module-paranamer 2.6.7
com.fasterxml.jackson.module jackson-module-scala_2.11 2.6.7.1
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.luben zstd-jni 1.3.2-2
com.github.rwl jtransforms 2.4.0
com.google.code.findbugs jsr305 2.0.1
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.h2database h2 1.3.174
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.microsoft.azure azure-data-lake-store-sdk 2.2.8
com.microsoft.azure azure-storage 5.2.0
com.microsoft.sqlserver mssql-jdbc 6.2.2.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.11 0.3
com.twitter chill-java 0.9.3
com.twitter chill_2.11 0.9.3
com.twitter parquet-hadoop-bundle 1.6.0
com.twitter util-app_2.11 6.23.0
com.twitter util-core_2.11 6.23.0
com.twitter util-jvm_2.11 6.23.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging-api_2.11 2.1.2
com.typesafe.scala-logging scala-logging-slf4j_2.11 2.1.2
com.univocity univocity-parsers 2.7.3
com.vlkan flatbuffers 1.2.0-3f79e055
com.zaxxer HikariCP 3.1.0
commons-beanutils commons-beanutils 1.9.3
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.7
io.airlift aircompressor 0.10
io.dropwizard.metrics metrics-core 3.1.5
io.dropwizard.metrics metrics-ganglia 3.1.5
io.dropwizard.metrics metrics-graphite 3.1.5
io.dropwizard.metrics metrics-healthchecks 3.1.5
io.dropwizard.metrics metrics-jetty9 3.1.5
io.dropwizard.metrics metrics-json 3.1.5
io.dropwizard.metrics metrics-jvm 3.1.5
io.dropwizard.metrics metrics-log4j 3.1.5
io.dropwizard.metrics metrics-servlets 3.1.5
io.netty netty 3.9.9.Final
io.netty netty-all 4.1.17.Final
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.validation validation-api 1.1.0.Final
javax.ws.rs javax.ws.rs-api 2.0.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.9.3
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
maven-trees hive-exec-with-glue hive-12679-patch_deploy
maven-trees hive-exec-with-glue hive-exec_shaded
net.hydromatic eigenbase-properties 1.1.5
net.razorvine pyrolite 4.13
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.5
net.snowflake snowflake-jdbc 3.6.15
net.snowflake spark-snowflake_2.11 2.4.10-spark_2.4
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt oncrpc 1.0.7
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr antlr4-runtime 4.7
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.10.0
org.apache.arrow arrow-memory 0.10.0
org.apache.arrow arrow-vector 0.10.0
org.apache.avro avro 1.8.2
org.apache.avro avro-ipc 1.8.2
org.apache.avro avro-mapred-hadoop2 1.8.2
org.apache.calcite calcite-avatica 1.2.0-incubating
org.apache.calcite calcite-core 1.2.0-incubating
org.apache.calcite calcite-linq4j 1.2.0-incubating
org.apache.commons commons-compress 1.8.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.5
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.12.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.3
org.apache.hadoop hadoop-auth 2.7.3
org.apache.hadoop hadoop-client 2.7.3
org.apache.hadoop hadoop-common 2.7.3
org.apache.hadoop hadoop-hdfs 2.7.3
org.apache.hadoop hadoop-mapreduce-client-app 2.7.3
org.apache.hadoop hadoop-mapreduce-client-common 2.7.3
org.apache.hadoop hadoop-mapreduce-client-core 2.7.3
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.3
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.3
org.apache.hadoop hadoop-yarn-api 2.7.3
org.apache.hadoop hadoop-yarn-client 2.7.3
org.apache.hadoop hadoop-yarn-common 2.7.3
org.apache.hadoop hadoop-yarn-server-common 2.7.3
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.4
org.apache.httpcomponents httpcore 4.4.8
org.apache.ivy ivy 2.4.0
org.apache.orc orc-core-nohive 1.5.2
org.apache.orc orc-mapreduce-nohive 1.5.2
org.apache.orc orc-shims 1.5.2
org.apache.parquet parquet-column 1.10.1.1-databricks3
org.apache.parquet parquet-common 1.10.1.1-databricks3
org.apache.parquet parquet-encoding 1.10.1.1-databricks3
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1.1-databricks3
org.apache.parquet parquet-jackson 1.10.1.1-databricks3
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.9.3
org.apache.xbean xbean-asm6-shaded 4.8
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.10
org.codehaus.janino janino 3.0.10
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.3.20.v20170531
org.eclipse.jetty jetty-continuation 9.3.20.v20170531
org.eclipse.jetty jetty-http 9.3.20.v20170531
org.eclipse.jetty jetty-io 9.3.20.v20170531
org.eclipse.jetty jetty-jndi 9.3.20.v20170531
org.eclipse.jetty jetty-plus 9.3.20.v20170531
org.eclipse.jetty jetty-proxy 9.3.20.v20170531
org.eclipse.jetty jetty-security 9.3.20.v20170531
org.eclipse.jetty jetty-server 9.3.20.v20170531
org.eclipse.jetty jetty-servlet 9.3.20.v20170531
org.eclipse.jetty jetty-servlets 9.3.20.v20170531
org.eclipse.jetty jetty-util 9.3.20.v20170531
org.eclipse.jetty jetty-webapp 9.3.20.v20170531
org.eclipse.jetty jetty-xml 9.3.20.v20170531
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.4.0-b34
org.glassfish.hk2 hk2-locator 2.4.0-b34
org.glassfish.hk2 hk2-utils 2.4.0-b34
org.glassfish.hk2 osgi-resource-locator 1.0.1
org.glassfish.hk2.external aopalliance-repackaged 2.4.0-b34
org.glassfish.hk2.external javax.inject 2.4.0-b34
org.glassfish.jersey.bundles.repackaged jersey-guava 2.22.2
org.glassfish.jersey.containers jersey-container-servlet 2.22.2
org.glassfish.jersey.containers jersey-container-servlet-core 2.22.2
org.glassfish.jersey.core jersey-client 2.22.2
org.glassfish.jersey.core jersey-common 2.22.2
org.glassfish.jersey.core jersey-server 2.22.2
org.glassfish.jersey.media jersey-media-jaxb 2.22.2
org.hibernate hibernate-validator 5.1.1.Final
org.iq80.snappy snappy 0.2
org.javassist javassist 3.18.1-GA
org.jboss.logging jboss-logging 3.1.3.GA
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.11 3.5.3
org.json4s json4s-core_2.11 3.5.3
org.json4s json4s-jackson_2.11 3.5.3
org.json4s json4s-scalap_2.11 3.5.3
org.lz4 lz4-java 1.4.0
org.mariadb.jdbc mariadb-java-client 2.1.2
org.mockito mockito-all 1.9.5
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.1.4
org.roaringbitmap RoaringBitmap 0.7.45
org.roaringbitmap shims 0.7.45
org.rocksdb rocksdbjni 5.2.1
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.11 2.11.12
org.scala-lang scala-library_2.11 2.11.12
org.scala-lang scala-reflect_2.11 2.11.12
org.scala-lang.modules scala-parser-combinators_2.11 1.1.0
org.scala-lang.modules scala-xml_2.11 1.0.5
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.11 1.12.5
org.scalactic scalactic_2.11 3.0.3
org.scalanlp breeze-macros_2.11 0.13.2
org.scalanlp breeze_2.11 0.13.2
org.scalatest scalatest_2.11 3.0.3
org.slf4j jcl-over-slf4j 1.7.16
org.slf4j jul-to-slf4j 1.7.16
org.slf4j slf4j-api 1.7.16
org.slf4j slf4j-log4j12 1.7.16
org.spark-project.hive hive-beeline 1.2.1.spark2
org.spark-project.hive hive-cli 1.2.1.spark2
org.spark-project.hive hive-jdbc 1.2.1.spark2
org.spark-project.hive hive-metastore 1.2.1.spark2
org.spark-project.spark unused 1.0.0
org.spire-math spire-macros_2.11 0.13.0
org.spire-math spire_2.11 0.13.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.5
org.typelevel machinist_2.11 0.6.1
org.typelevel macro-compat_2.11 1.1.1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.7.3
org.yaml snakeyaml 1.16
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52