Databricks Runtime 8.3 and Databricks Runtime 8.3 Photon

Databricks released these images in June 2021.

The following release notes provide information about Databricks Runtime 8.3 and Databricks Runtime 8.3 Photon, powered by Apache Spark 3.1.1.

New features and improvements

Accelerate SQL workloads with Photon (Public Preview)

Photon is the new native vectorized engine on Databricks, directly compatible with Apache Spark APIs. It is included as part of a new high-performance runtime designed to run your SQL workloads faster and reduce your total cost per workload. This runtime is a variant of the standard Databricks Runtime and is selectable when you create clusters in the UI, the Clusters API, or the Jobs API (specified in the APIs as 8.3.x-photon-scala2.12).

Photon supports a limited set of instance types on the driver and worker nodes. Instance types with Photon consume DBUs at a different rate than the same instance type running a non-Photon runtime. For more information about Photon instances and DBU consumption, see the Databricks pricing page.

Generated columns in Delta tables (Public Preview)

Delta Lake now supports generated columns, which are a special type of column whose values are automatically generated based on a user-specified function over other columns in the Delta table. You can use most built-in SQL functions to generate the values of these generated columns. For example, you can automatically generate a date column (for partitioning the table by date) from the timestamp column; any writes into the table need only specify the data for the timestamp column. You can create Delta tables with generated columns using SQL, Scala, Java or Python APIs.

For more information, see Use generated columns.

Auto Loader features and improvements

Schema inference for CSV files in Auto Loader

Auto Loader now supports schema inference and evolution on CSV files. Auto Loader provides the following capabilities on top of the existing CSV parser in Apache Spark:

  • Schema merging: Auto Loader can ingest CSV files that have different schema (different number of columns, different ordering of columns) across files.
  • Rescued data column: You can use the rescued data column to rescue unexpected data that may appear in your CSV files. This includes data that can not be parsed in the data type that’s expected, columns that have a different casing or null values in the header, or additional columns that were not part of the expected schema.

For details, see Schema inference and evolution in Auto Loader.

Improved startup time for Auto Loader streams

Auto Loader streams now perform the initial backfill for the stream asynchronously when starting for the first time, leading to a much faster start up time for the stream. This can allow you to iterate quickly on your code with production data, especially when you have to ingest data from directories that contain millions or billions of files.

In addition, the bootstrap time of streams that are restarting are also improved, because we have parallelized the download and upload of the RocksDB files that Auto Loader leverages for providing exactly-once semantics.

Faster directory listing in Auto Loader

We greatly improved the efficiency of directory listing in Auto Loader. A side effect of this performance improvement is that the stream may emit more list requests to the storage system when there is no new data to process, which can lead to a surge on list request charges. As a general best practice, Databricks recommends that you set a reasonable trigger interval for production streaming pipelines. See Structured Streaming in production.

Reduced storage overhead for Auto Loader checkpoints

Auto Loader streams now automatically clean up stale files in the checkpoint directory asynchronously to keep the checkpoint directory size from growing indefinitely and reduce storage costs.

Auto Loader includes the file path in the rescued data column when available

The rescued data column automatically provides the file path of the rescued data when applicable in a column named _file_ path. This can help you track down the root cause of data quality issues. The column is not included if the data schema contains a column called _file_path. You can use the SQL configuration spark.databricks.sql.rescuedDataColumn.filePath.name to rename the column if necessary.

Auto Loader supports file renames in Azure Data Lake Storage Gen2 in file notification mode

Auto Loader now supports BlobRenamed events for Azure Data Lake Storage Gen2 when running in file notification mode. To process files that are uploaded to an Azure Data Lake Storage Gen2 container through a rename operation with file notifications, start a new stream with Auto Loader using Databricks Runtime 8.3. To ensure that a file is processed exactly once, make sure that the source directory that the file is being renamed from is not watched by Auto Loader.

Create Delta tables with new programmatic APIs (Public Preview)

You can now create new Delta tables programmatically (using Scala, Java, and Python) without using DataFrame APIs. New DeltaTableBuilder and DeltaColumnBuilder APIs let you specify all of the table details that you can specify using SQL CREATE TABLE.

For more information, see Create a table.

Correct calculation of Delta table sizes in SQL ANALYZE

Existing analyze logic incorrectly computes the table size for Delta tables and updates the catalog with incorrect size. The fix is to get the size of a Delta table from the Delta log.

Detailed metrics of RocksDB performance when using RocksDBStateStore

If you have configured your Structured Streaming query to use RocksDB as the state store, you can now get better visibility into the performance of RocksDB, with detailed metrics on get/put latencies, compaction latencies, cache hits, and so on. These metrics are available through the StreamingQueryProgress and StreamingQueryListener APIs for monitoring a streaming query.

For more information, see RocksDB state store metrics.

Automatic Optimized Writes

Optimized Writes on partitioned Delta tables are now automatically enabled for update and delete queries containing subqueries.

Enable bucketed joins if only one join side is bucketed

A new config spark.databricks.sql.minBucketsForBucketedJoin enables a bucketed join if only one join side is bucketed and the number of buckets is not less than this config value. By default, this config value is the same as the default shuffle partitions number (200).

Improved security when defining Spark UDFs (Public Preview)

User info functions current_user and is_member can no longer be overridden by temporary functions including Python spark.udf.register or SQL create or replace temp function.

CSE-KMS endpoint supported in the new S3 client

The S3 connector now supports the fs.s3a.cse.kms.endpoint option to configure a custom KMS endpoint when using AWS S3 client-side encryption with keys managed by AWS KMS service (CSE-KMS). See the AWS article AWS Key Management Service endpoints and quotas for the list of valid endpoints. If the option is not specified, CSE-KMS defaults to us-east-1 region by default. If you use S3 client-side encryption with a customer-provided key, this option has no effect.

Easy migration of OSS Hadoop configurations to Databricks clusters, thanks to support for both OSS and shaded class names for S3 AWS credentials providers

Open-source Hadoop AWS credentials providers, for example, org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider, org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider, and org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider, can be configured as part of Hadoop configuration for S3 Storage Connector in addition to existing authentication methods. This enables easy migration of OSS Hadoop configuration to a Databricks cluster.

Reduced number of requests to schema registry for queries with from_avro

Queries with from_avro with schema registry support no longer generate as many requests to the schema registry service, saving operational cost.

Multiple results in R with ListResults (Public Preview)

Databricks R notebooks now support multiple results in each cell. Previously only a single result was rendered for each notebook cell. Currently the results of a single cell in R notebooks are displayed in the following order:

  1. RShiny URL
  2. Plot
  3. displayHTML outputs
  4. Tables
  5. stdout

Bug fixes

  • Disabled a list of pushed down predicates (StartsWith, EndsWith, Contains, Not(EqualTo()), and DataType) for AWS Glue Catalog since they are not supported in Glue yet.

Library upgrades

  • Upgraded Python library:
    • koalas upgraded from 1.7.0 to 1.8.0.
    • pandas upgraded from 1.1.3 to 1.1.5.
    • s3transfer upgraded from 0.3.4 to 0.3.6.
  • Upgraded R library:
    • SparkR upgraded from 3.1.1 to 3.1.2.
  • Upgraded Java library:
    • mariadb-java-client from 2.1.2 to 2.2.5.
    • parquet-column from 1.10.1-databricks6 to 1.10.1-databricks9
    • parquet-common from 1.10.1-databricks6 to 1.10.1-databricks9
    • parquet-encoding from 1.10.1-databricks6 to 1.10.1-databricks9
    • parquet-hadoop from 1.10.1-databricks6 to 1.10.1-databricks9
    • parquet-jackson from 1.10.1-databricks6 to 1.10.1-databricks9

Apache Spark

Databricks Runtime 8.3 includes Apache Spark 3.1.1. This release includes all Spark fixes and improvements included in Databricks Runtime 8.2, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-34614] [SQL] ANSI mode: Casting String to Boolean should throw exception on parse error
  • [SPARK-34246] [FOLLOWUP] Change the definition of `findTightestCommonT…
  • [SPARK-35213] [SQL] Keep the correct ordering of nested structs in chained withField operations
  • [SPARK-35096] [SQL] SchemaPruning should adhere spark.sql.caseSensitive config
  • [SPARK-35227] [BUILD] Update the resolver for spark-packages in SparkSubmit
  • [SPARK-35224] [SQL] Fix buffer overflow in MutableProjectionSuite
  • [SPARK-34245] [CORE] Ensure Master removes executors that failed to send finished state
  • [SPARK-34856] [SQL] ANSI mode: Allow casting complex types as string type
  • [SPARK-34946] [SQL] Block unsupported correlated scalar subquery in Aggregate
  • [SPARK-35014] Fix the PhysicalAggregation pattern to not rewrite foldable expressions
  • [SPARK-34769] [SQL] AnsiTypeCoercion: return closest con…

System environment

  • Operating System: Ubuntu 18.04.5 LTS
  • Java: Zulu 8.52.0.23-CA-linux64 (build 1.8.0_282-b08)
  • Scala: 2.12.10
  • Python: 3.8.8
  • R: R version 4.0.4 (2021-02-15)
  • Delta Lake 1.0.0

Installed Python libraries

Library Version Library Version Library Version
appdirs 1.4.4 asn1crypto 1.4.0 backcall 0.2.0
boto3 1.16.7 botocore 1.19.7 brotlipy 0.7.0
certifi 2020.12.5 cffi 1.14.3 chardet 3.0.4
cryptography 3.1.1 cycler 0.10.0 Cython 0.29.21
decorator 4.4.2 distlib 0.3.1 docutils 0.15.2
entrypoints 0.3 facets-overview 1.0.0 filelock 3.0.12
idna 2.10 ipykernel 5.3.4 ipython 7.19.0
ipython-genutils 0.2.0 jedi 0.17.2 jmespath 0.10.0
joblib 0.17.0 jupyter-client 6.1.7 jupyter-core 4.6.3
kiwisolver 1.3.0 koalas 1.8.0 matplotlib 3.2.2
numpy 1.19.2 pandas 1.1.5 parso 0.7.0
patsy 0.5.1 pexpect 4.8.0 pickleshare 0.7.5
pip 20.2.4 plotly 4.14.3 prompt-toolkit 3.0.8
protobuf 3.17.0 psycopg2 2.8.5 ptyprocess 0.6.0
pyarrow 1.0.1 pycparser 2.20 Pygments 2.7.2
pyOpenSSL 19.1.0 pyparsing 2.4.7 PySocks 1.7.1
python-dateutil 2.8.1 pytz 2020.5 pyzmq 19.0.2
requests 2.24.0 retrying 1.3.3 s3transfer 0.3.6
scikit-learn 0.23.2 scipy 1.5.2 seaborn 0.10.0
setuptools 50.3.1 six 1.15.0 statsmodels 0.12.0
threadpoolctl 2.1.0 tornado 6.0.4 traitlets 5.0.5
urllib3 1.25.11 virtualenv 20.2.1 wcwidth 0.2.5
wheel 0.35.1        

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.2.1
base 4.0.4 base64enc 0.1-3 BH 1.72.0-3
bit 4.0.4 bit64 4.0.5 blob 1.2.1
boot 1.3-27 brew 1.0-6 brio 1.1.0
broom 0.7.2 callr 3.5.1 caret 6.0-86
cellranger 1.1.0 chron 2.3-56 class 7.3-18
cli 2.2.0 clipr 0.7.1 cluster 2.1.1
codetools 0.2-18 colorspace 2.0-0 commonmark 1.7
compiler 4.0.4 config 0.3 covr 3.5.1
cpp11 0.2.4 crayon 1.3.4 credentials 1.3.0
crosstalk 1.1.0.1 curl 4.3 data.table 1.13.4
datasets 4.0.4 DBI 1.1.0 dbplyr 2.0.0
desc 1.2.0 devtools 2.3.2 diffobj 0.3.2
digest 0.6.27 dplyr 1.0.2 DT 0.16
ellipsis 0.3.1 evaluate 0.14 fansi 0.4.1
farver 2.0.3 fastmap 1.0.1 forcats 0.5.0
foreach 1.5.1 foreign 0.8-81 forge 0.2.0
fs 1.5.0 future 1.21.0 generics 0.1.0
gert 1.0.2 ggplot2 3.3.2 gh 1.2.0
gitcreds 0.1.1 glmnet 4.0-2 globals 0.14.0
glue 1.4.2 gower 0.2.2 graphics 4.0.4
grDevices 4.0.4 grid 4.0.4 gridExtra 2.3
gsubfn 0.7 gtable 0.3.0 haven 2.3.1
highr 0.8 hms 0.5.3 htmltools 0.5.0
htmlwidgets 1.5.3 httpuv 1.5.4 httr 1.4.2
hwriter 1.3.2 hwriterPlus 1.0-3 ini 0.3.1
ipred 0.9-9 isoband 0.2.3 iterators 1.0.13
jsonlite 1.7.2 KernSmooth 2.23-18 knitr 1.30
labeling 0.4.2 later 1.1.0.1 lattice 0.20-41
lava 1.6.8.1 lazyeval 0.2.2 lifecycle 0.2.0
listenv 0.8.0 lubridate 1.7.9.2 magrittr 2.0.1
markdown 1.1 MASS 7.3-53.1 Matrix 1.3-2
memoise 1.1.0 methods 4.0.4 mgcv 1.8-33
mime 0.9 ModelMetrics 1.2.2.2 modelr 0.1.8
munsell 0.5.0 nlme 3.1-152 nnet 7.3-15
numDeriv 2016.8-1.1 openssl 1.4.3 parallel 4.0.4
parallelly 1.22.0 pillar 1.4.7 pkgbuild 1.1.0
pkgconfig 2.0.3 pkgload 1.1.0 plogr 0.2.0
plyr 1.8.6 praise 1.0.0 prettyunits 1.1.1
pROC 1.16.2 processx 3.4.5 prodlim 2019.11.13
progress 1.2.2 promises 1.1.1 proto 1.0.0
ps 1.5.0 purrr 0.3.4 r2d3 0.2.3
R6 2.5.0 randomForest 4.6-14 rappdirs 0.3.1
rcmdcheck 1.3.3 RColorBrewer 1.1-2 Rcpp 1.0.5
readr 1.4.0 readxl 1.3.1 recipes 0.1.15
rematch 1.0.1 rematch2 2.1.2 remotes 2.2.0
reprex 0.3.0 reshape2 1.4.4 rex 1.2.0
rlang 0.4.9 rmarkdown 2.6 RODBC 1.3-17
roxygen2 7.1.1 rpart 4.1-15 rprojroot 2.0.2
Rserve 1.8-7 RSQLite 2.2.1 rstudioapi 0.13
rversions 2.0.2 rvest 0.3.6 scales 1.1.1
selectr 0.4-2 sessioninfo 1.1.1 shape 1.4.5
shiny 1.5.0 sourcetools 0.1.7 sparklyr 1.5.2
SparkR 3.1.2 spatial 7.3-11 splines 4.0.4
sqldf 0.4-11 SQUAREM 2020.5 stats 4.0.4
stats4 4.0.4 stringi 1.5.3 stringr 1.4.0
survival 3.2-7 sys 3.4 tcltk 4.0.4
TeachingDemos 2.10 testthat 3.0.0 tibble 3.0.4
tidyr 1.1.2 tidyselect 1.1.0 tidyverse 1.3.0
timeDate 3043.102 tinytex 0.28 tools 4.0.4
usethis 2.0.0 utf8 1.1.4 utils 4.0.4
uuid 0.1-4 vctrs 0.3.5 viridisLite 0.3.0
waldo 0.2.3 whisker 0.4 withr 2.3.0
xfun 0.19 xml2 1.3.2 xopen 1.0.0
xtable 1.8-4 yaml 2.2.1 zip 2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.11.655
com.amazonaws aws-java-sdk-cloudformation 1.11.655
com.amazonaws aws-java-sdk-cloudfront 1.11.655
com.amazonaws aws-java-sdk-cloudhsm 1.11.655
com.amazonaws aws-java-sdk-cloudsearch 1.11.655
com.amazonaws aws-java-sdk-cloudtrail 1.11.655
com.amazonaws aws-java-sdk-cloudwatch 1.11.655
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.655
com.amazonaws aws-java-sdk-codedeploy 1.11.655
com.amazonaws aws-java-sdk-cognitoidentity 1.11.655
com.amazonaws aws-java-sdk-cognitosync 1.11.655
com.amazonaws aws-java-sdk-config 1.11.655
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-datapipeline 1.11.655
com.amazonaws aws-java-sdk-directconnect 1.11.655
com.amazonaws aws-java-sdk-directory 1.11.655
com.amazonaws aws-java-sdk-dynamodb 1.11.655
com.amazonaws aws-java-sdk-ec2 1.11.655
com.amazonaws aws-java-sdk-ecs 1.11.655
com.amazonaws aws-java-sdk-efs 1.11.655
com.amazonaws aws-java-sdk-elasticache 1.11.655
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.655
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.655
com.amazonaws aws-java-sdk-elastictranscoder 1.11.655
com.amazonaws aws-java-sdk-emr 1.11.655
com.amazonaws aws-java-sdk-glacier 1.11.655
com.amazonaws aws-java-sdk-iam 1.11.655
com.amazonaws aws-java-sdk-importexport 1.11.655
com.amazonaws aws-java-sdk-kinesis 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-lambda 1.11.655
com.amazonaws aws-java-sdk-logs 1.11.655
com.amazonaws aws-java-sdk-machinelearning 1.11.655
com.amazonaws aws-java-sdk-opsworks 1.11.655
com.amazonaws aws-java-sdk-rds 1.11.655
com.amazonaws aws-java-sdk-redshift 1.11.655
com.amazonaws aws-java-sdk-route53 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-ses 1.11.655
com.amazonaws aws-java-sdk-simpledb 1.11.655
com.amazonaws aws-java-sdk-simpleworkflow 1.11.655
com.amazonaws aws-java-sdk-sns 1.11.655
com.amazonaws aws-java-sdk-sqs 1.11.655
com.amazonaws aws-java-sdk-ssm 1.11.655
com.amazonaws aws-java-sdk-storagegateway 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws aws-java-sdk-support 1.11.655
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.655
com.amazonaws jmespath-java 1.11.655
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.fasterxml.jackson.datatype jackson-datatype-joda 2.10.0
com.fasterxml.jackson.module jackson-module-paranamer 2.10.0
com.fasterxml.jackson.module jackson-module-scala_2.12 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.joshelser dropwizard-metrics-hadoop-metrics2-reporter 0.1.2
com.github.luben zstd-jni 1.4.8-1
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.flatbuffers flatbuffers-java 1.9.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 1.4.195
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.9.5
com.twitter chill_2.12 0.9.5
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 3.1.0
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
hive-2.3__hadoop-2.7 jets3t-0.7 liball_deps_2.12
hive-2.3__hadoop-2.7 zookeeper-3.4 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.10
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.51.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.5
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
maven-trees hive-2.3__hadoop-2.7 liball_deps_2.12
net.razorvine pyrolite 4.30
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.12.8
net.snowflake spark-snowflake_2.12 2.8.1-spark_3.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8-1
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 2.0.0
org.apache.arrow arrow-memory-core 2.0.0
org.apache.arrow arrow-memory-netty 2.0.0
org.apache.arrow arrow-vector 2.0.0
org.apache.avro avro 1.8.2
org.apache.avro avro-ipc 1.8.2
org.apache.avro avro-mapred-hadoop2 1.8.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.10
org.apache.commons commons-math3 3.4.1
org.apache.commons commons-text 1.6
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.12.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-client 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.hadoop hadoop-hdfs 2.7.4
org.apache.hadoop hadoop-mapreduce-client-app 2.7.4
org.apache.hadoop hadoop-mapreduce-client-common 2.7.4
org.apache.hadoop hadoop-mapreduce-client-core 2.7.4
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.4
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.4
org.apache.hadoop hadoop-yarn-api 2.7.4
org.apache.hadoop hadoop-yarn-client 2.7.4
org.apache.hadoop hadoop-yarn-common 2.7.4
org.apache.hadoop hadoop-yarn-server-common 2.7.4
org.apache.hive hive-beeline 2.3.7
org.apache.hive hive-cli 2.3.7
org.apache.hive hive-common 2.3.7
org.apache.hive hive-exec-core 2.3.7
org.apache.hive hive-jdbc 2.3.7
org.apache.hive hive-llap-client 2.3.7
org.apache.hive hive-llap-common 2.3.7
org.apache.hive hive-metastore 2.3.7
org.apache.hive hive-serde 2.3.7
org.apache.hive hive-shims 2.3.7
org.apache.hive hive-storage-api 2.7.2
org.apache.hive hive-vector-code-gen 2.3.7
org.apache.hive.shims hive-shims-0.23 2.3.7
org.apache.hive.shims hive-shims-common 2.3.7
org.apache.hive.shims hive-shims-scheduler 2.3.7
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.12
org.apache.ivy ivy 2.4.0
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.5.12
org.apache.orc orc-mapreduce 1.5.12
org.apache.orc orc-shims 1.5.12
org.apache.parquet parquet-column 1.10.1-databricks9
org.apache.parquet parquet-common 1.10.1-databricks9
org.apache.parquet parquet-encoding 1.10.1-databricks9
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1-databricks9
org.apache.parquet parquet-jackson 1.10.1-databricks9
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.velocity velocity 1.5
org.apache.xbean xbean-asm7-shaded 4.15
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.4.14
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.36.v20210114
org.eclipse.jetty jetty-continuation 9.4.36.v20210114
org.eclipse.jetty jetty-http 9.4.36.v20210114
org.eclipse.jetty jetty-io 9.4.36.v20210114
org.eclipse.jetty jetty-jndi 9.4.36.v20210114
org.eclipse.jetty jetty-plus 9.4.36.v20210114
org.eclipse.jetty jetty-proxy 9.4.36.v20210114
org.eclipse.jetty jetty-security 9.4.36.v20210114
org.eclipse.jetty jetty-server 9.4.36.v20210114
org.eclipse.jetty jetty-servlet 9.4.36.v20210114
org.eclipse.jetty jetty-servlets 9.4.36.v20210114
org.eclipse.jetty jetty-util 9.4.36.v20210114
org.eclipse.jetty jetty-util-ajax 9.4.36.v20210114
org.eclipse.jetty jetty-webapp 9.4.36.v20210114
org.eclipse.jetty jetty-xml 9.4.36.v20210114
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.30
org.glassfish.jersey.containers jersey-container-servlet-core 2.30
org.glassfish.jersey.core jersey-client 2.30
org.glassfish.jersey.core jersey-common 2.30
org.glassfish.jersey.core jersey-server 2.30
org.glassfish.jersey.inject jersey-hk2 2.30
org.glassfish.jersey.media jersey-media-jaxb 2.30
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M5
org.json4s json4s-core_2.12 3.7.0-M5
org.json4s json4s-jackson_2.12 3.7.0-M5
org.json4s json4s-scalap_2.12 3.7.0-M5
org.lz4 lz4-java 1.7.1
org.mariadb.jdbc mariadb-java-client 2.2.5
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.1.4
org.roaringbitmap RoaringBitmap 0.9.0
org.roaringbitmap shims 0.9.0
org.rocksdb rocksdbjni 6.2.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.10
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-collection-compat_2.12 2.1.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.0
org.scalanlp breeze_2.12 1.0
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.30
org.slf4j jul-to-slf4j 1.7.30
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-log4j12 1.7.30
org.spark-project.spark unused 1.0.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.5
org.typelevel algebra_2.12 2.0.0-M2
org.typelevel cats-kernel_2.12 2.0.0-M4
org.typelevel machinist_2.12 0.6.8
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0-M1
org.typelevel spire-platform_2.12 0.17.0-M1
org.typelevel spire-util_2.12 0.17.0-M1
org.typelevel spire_2.12 0.17.0-M1
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.2
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52
antlr antlr 2.7.7
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws jmespath-java 1.11.655
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware.kryo kryo 2.21
com.esotericsoftware.minlog minlog 1.2
com.esotericsoftware.reflectasm reflectasm-shaded 1.07
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.helger profiler 1.1.1
com.jolbox bonecp 0.8.0.RELEASE
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.thoughtworks.paranamer paranamer 2.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter parquet-hadoop-bundle 1.3.2
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-codec commons-codec 1.8
commons-collections commons-collections 3.2.2
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-io commons-io 2.5
commons-lang commons-lang 2.6
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
hive-2.3__hadoop-2.7 jets3t-0.7 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty 3.8.0.Final
io.netty netty-all 4.1.51.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
jakarta.validation jakarta.validation-api 2.0.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javolution javolution 5.5.1
jline jline 0.9.94
joda-time joda-time 2.10.5
junit junit 3.8.1
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
log4j log4j 1.2.17
net.sf.jpam jpam 1.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.8.2
org.apache.avro avro 1.8.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-compress 1.9
org.apache.commons commons-lang3 3.4
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.10.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.4.1
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.1
org.apache.httpcomponents httpcore 4.4.12
org.apache.thrift libfb303 0.9.0
org.apache.thrift libthrift 0.9.2
org.apache.velocity velocity 1.5
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.groovy groovy-all 2.1.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.4.36.v20210114
org.eclipse.jetty jetty-continuation 9.4.36.v20210114
org.eclipse.jetty jetty-http 9.4.36.v20210114
org.eclipse.jetty jetty-io 9.4.36.v20210114
org.eclipse.jetty jetty-proxy 9.4.36.v20210114
org.eclipse.jetty jetty-security 9.4.36.v20210114
org.eclipse.jetty jetty-server 9.4.36.v20210114
org.eclipse.jetty jetty-servlet 9.4.36.v20210114
org.eclipse.jetty jetty-servlets 9.4.36.v20210114
org.eclipse.jetty jetty-util 9.4.36.v20210114
org.eclipse.jetty jetty-util-ajax 9.4.36.v20210114
org.hibernate.validator hibernate-validator 6.1.0.Final
org.iq80.snappy snappy 0.2
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.json json 20090211
org.objenesis objenesis 1.2
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scalactic scalactic_2.12 3.0.8
org.scalatest scalatest_2.12 3.0.8
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-api 1.7.5
org.slf4j slf4j-log4j12 1.7.30
org.slf4j slf4j-log4j12 1.7.5
org.spark-project.hive hive-ant 0.13.1a
org.spark-project.hive hive-beeline 0.13.1a
org.spark-project.hive hive-cli 0.13.1a
org.spark-project.hive hive-common 0.13.1a
org.spark-project.hive hive-exec 0.13.1a
org.spark-project.hive hive-jdbc 0.13.1a
org.spark-project.hive hive-metastore 0.13.1a
org.spark-project.hive hive-serde 0.13.1a
org.spark-project.hive hive-service 0.13.1a
org.spark-project.hive hive-shims 0.13.1a
org.spark-project.hive.shims hive-shims-0.20 0.13.1a
org.spark-project.hive.shims hive-shims-0.20S 0.13.1a
org.spark-project.hive.shims hive-shims-0.23 0.13.1a
org.spark-project.hive.shims hive-shims-common 0.13.1a
org.spark-project.hive.shims hive-shims-common-secure 0.13.1a
org.spark-project.protobuf protobuf-java 2.5.0-spark
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.5
org.tukaani xz 1.5
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial.snappy snappy-java 1.1.2.6
org.xerial.snappy snappy-java 1.1.8.2
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52