Databricks Runtime 8.4 and Databricks Runtime 8.4 Photon

Databricks released these images in July 2021.

The following release notes provide information about Databricks Runtime 8.4 and Databricks Runtime 8.4 Photon, powered by Apache Spark 3.1.2.

New features and improvements

Improve cluster security by using credential passthrough with AWS Glue Data catalog

Credential passthrough is now supported when using AWS Glue Data Catalog as the external Hive metastore. Callers with different permissions can write to the AWS Glue Data Catalog from the same cluster. For more information, see Step 6: Launch a cluster with the Glue Catalog instance profile.

Delta Lake features and improvements

Delta table change data feed (GA)

The Delta table change data feed is now generally available. It represents the row level changes between different versions of the table. When enabled, additional information is recorded regarding row level changes for every write operation on the table. See Change data feed.

Load shared Delta tables easily with Databricks Runtime

The Apache Spark Connector for Delta Sharing 0.1.0 is now in Databricks Runtime. You can load a shared table using spark.read.format("deltaSharing").load(uri) directly without attaching the Delta Sharing Spark connector to your cluster.

More tables benefit from dynamic file pruning

The dynamic file pruning feature has been tuned to trigger on tables with fewer files. See Dynamic file pruning.

Better performance with automatic target file size tuning

The target file size for Delta tables is now automatically tuned based on the table size. Previously, the target file size for OPTIMIZE and OPTIMIZE ZORDER BY was 1GB. With autotuning based on table size, Delta tables up to 2.56TB will use 256MB as the target size. Tables larger than 10TB will use 1GB as before. Tables between these sizes will use target sizes that grow proportionally with the table size. See Tune file size.

More ways to specify tables in DeltaTable.forName

DeltaTable.forName now supports using delta.`<path>` to identify tables.

Robust streaming multi-table writes using foreachBatch

Idempotent Delta streaming writes within the foreachBatch() command are now supported. For details, see Idempotent multi-table writes.

Improved read query performance in certain workloads due to tuned checkpoints

Delta Lake now tunes how frequently it does enhanced checkpoints. Instead of checkpointing at a fixed interval, Delta now dynamically adjusts the checkpoint frequency based on certain event triggers. This improves read query performance in workloads where some data skipping optimization couldn’t be applied before. To use these optimizations, upgrade your jobs that write to Delta Lake to Databricks Runtime 8.4. See Enhanced checkpoints for low-latency queries.

Create GroupState to test user defined Structured Streaming functions

Until now, only the Structured Streaming engine could create instances of GroupState. Hence, any unit tests of the user-defined function required running a streaming query in Apache Spark.

Now you can create instances of GroupState using TestGroupState.create(…). This allow you to test a user-defined function in simple unit tests that do not require running Spark. See Test state update function. Specifically, it produces instances of type TestGroupState that extends the interface GroupState with additional methods for introspecting the internal state after the user-defined function has been applied.

Auto Loader features and improvements

Configure backfilling to capture missed files

Auto Loader now supports performing backfills asynchronously to capture any files that could have been missed with file notifications. File storage systems and notification systems cannot guarantee 100% delivery of all file events. Therefore Databricks recommends enabling periodic backfills to capture all of your data with Auto Loader. Use the cloudFiles.backfillInterval option to schedule regular backfills over your data. See S3 Common Auto Loader options and ADLS Gen2 Common Auto Loader options.

Cross-account file notification access

Auto Loader now supports loading data across AWS accounts by using assuming a role. After setting the temporary security credentials created by AssumeRole, you can have Auto Loader load cloud files cross-accounts. See Securely ingest data in a different AWS account

Bound storage footprint for large volume streams

You can now configure Auto Loader to expire and remove entries in RocksDB to bound its storage footprint in the checkpoint location. Databricks does not recommend that you use unless you are ingesting data in the order of millions of files an hour. Setting this option incorrectly or attempting to tune it can lead to many data quality issues, such as unprocessed files being ignored or duplicate of some files instead of exactly once processing. For details, see How to choose maxFileAge for S3, How to choose maxFileAge for Azure Data Lake Storage Gen2, and How to choose maxFileAge for GCS.

Simplified configuration with pathless support

S3 buckets

You can now provide the SQS queue that receives events from multiple paths or S3 buckets. If you provide the SQS queue URL, the path option is not required for this use case. Auto Loader constructs S3 paths using the bucket and key from the S3 events. If you want to read the files through DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes to DBFS. This is not required unless you’re accessing data in different accounts with AssumeRole.

See File notification options.

Azure Data Lake Storage Gen2 containers

You can now provide the Azure queue that receives events from multiple containers. If you provide the Azure queue name, the path option is not required. By default, Auto Loader constructs Azure Data Lake Storage Gen2 paths using the container and key in the file events. If you want to use WASB paths or DBFS mount points, you can use cloudFiles.pathRewrites to change path prefixes.

See File notification options.

Connector upgrades

  • The Snowflake Spark connector has been updated to v2.9.0.
  • KMS encryption is now supported in the UNLOAD statement of the Redshift connector.

Bug fixes

  • Fixed an issue for clusters enabled with table access control where select * from folder can show folder content even if the user has no file access permission.
  • Non-admin database owners are now able to drop non-owned tables in databases. This fixes the issue where database owners could not drop a database if non-owned tables existed in the database.

Library upgrades

  • Upgraded Python libraries:
    • certifi upgraded from 2020.12.5 to 2021.5.30
    • distill upgraded from 0.3.1 to 0.3.2
    • koalas upgraded from 1.8.0 to 1.8.1
    • protobuf upgraded from 3.17.0 to 3.17.3
  • Upgraded R libraries:
    • base from 4.0.4 to 4.1.0
    • boot from 1.3-27 to 1.3-28
    • class from 7.3-18 to 7.3-19
    • cluster from 2.1.1 to 2.1.2
    • compiler from 4.0.4 to 4.1.0
    • datasets from 4.0.4 to 4.1.0
    • graphics from 4.0.4 to 4.1.0
    • grDevices from 4.0.4 to 4.1.0
    • grid from 4.0.4 to 4.1.0
    • KernSmooth from 2.23-18 to 2.23-20
    • lattice from 0.20-41 to 0.20-44
    • MASS from 7.3-53.1 to 7.3-54
    • Matrix from 1.3-2 to 1.3-3
    • methods from 4.0.4 to 4.1.0
    • mgcv from 1.8-33 to 1.8-35
    • nnet from 7.3-15 to 7.3-16
    • parallel from 4.0.4 to 4.1.0
    • Rserve from 1.8-7 to 1.8-8
    • SparkR from 3.1.1 to 3.1.2
    • splines from 4.0.4 to 4.1.0
    • stats from 4.0.4 to 4.1.0
    • stats4 from 4.0.4 to 4.1.0
    • survival from 3.2-7 to 3.2-11
    • tcltk from 4.0.4 to 4.1.0
    • tools from 4.0.4 to 4.1.0
    • utils from 4.0.4 to 4.1.0
  • Upgraded Java libraries:
    • snowflake-jdbc from 3.12.8 to 3.13.3
    • spark-snowflake_2.12 from 2.8.1-spark_3.0 to 2.9.0-spark_3.1
    • RoaringBitmap from 0.9.0 to 0.9.14
    • shims from 0.9.0 to 0.9.14
    • rocksdbjni fromm 6.2.2 to 6.20.3

Apache Spark

Databricks Runtime 8.4 includes Apache Spark 3.1.2. This release includes all Spark fixes and improvements included in Databricks Runtime 8.3 and Databricks Runtime 8.3 Photon, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-35792] [SQL] View should not capture configs used in RelationConversions
  • [SPARK-35700] [SQL] Read char/varchar orc table with created and written by external systems
  • [SPARK-35636] [SQL] Lambda keys should not be referenced outside of the lambda function
  • [SPARK-35800] [Cherry Pick] Improving GroupState testability by introducing TestGroupState
  • [SPARK-35391] Fix memory leak in ExecutorAllocationListener
  • [SPARK-35799] [CherryPick] Fix the allUpdatesTimeMs metric measuring in FlatMapGroupsWithStateExec
  • [SPARK-35763] [SS] Remove the StateStoreCustomMetric subclass enumeration dependency
  • [SPARK-35791 [SQL] Release on-going map properly for NULL-aware ANTI join
  • [SPARK-35695] [SQL] Collect observed metrics from cached and adaptive execution sub-trees
  • [SPARK-35767] [SQL] Avoid executing child plan twice in CoalesceExec
  • [SPARK-35746] [UI] Fix taskid in the stage page task event timeline
  • [SPARK-35673] [SQL] Fix user-defined hint and unrecognized hint in subquery.
  • [SPARK-35714] [CORE] Bug fix for deadlock during the executor shutdown
  • [SPARK-35689] [SS] Add log warn when keyWithIndexToValue returns null value
  • [SPARK-35589] [CORE][3.1] BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
  • [SPARK-35643] [PYTHON] Fix ambiguous reference in functions.py column()
  • [SPARK-35652] [SQL] joinWith on two table generated from same one
  • [SPARK-35679] [SQL] instantToMicros overflow
  • [SPARK-35602] [SS] Update state schema to be able to accept long length JSON
  • [SPARK-35653] [SQL] Fix CatalystToExternalMap interpreted path fails for Map with case classes as keys or values
  • [SPARK-35296] [SQL] Allow Dataset.observe to work even if CollectMetricsExec in a task handles multiple partitions.
  • [SPARK-35659] [SS] Avoid write null to StateStore
  • [SPARK-35665] [SQL] Resolve UnresolvedAlias in CollectMetrics
  • [SPARK-35558] Optimizes for multi-quantile retrieval
  • [SPARK-35621] [SQL] Add rule id pruning to the TypeCoercion rule
  • [SPARK-35077] [SQL] Migrate to transformWithPruning for leftover optimizer rules
  • [SPARK-35610] [CORE] Fix the memory leak introduced by the Executor’s stop shutdown hook
  • [SPARK-35544] [SQL] Add tree pattern pruning to Analyzer rules
  • [SPARK-35566] [SS] Fix StateStoreRestoreExec output rows
  • [SPARK-35454] [SQL][3.1] One LogicalPlan can match multiple dataset ids
  • [SPARK-35538] [SQL] Migrate transformAllExpressions call sites to use transformAllExpressionsWithPruning
  • [SPARK-35106] [Core][SQL] Avoid failing rename caused by destination directory not exist
  • [SPARK-35287] [SQL] Allow RemoveRedundantProjects to preserve ProjectExec which generates UnsafeRow for DataSourceV2ScanRelation
  • [SPARK-35495] [R] Change SparkR maintainer for CRAN
  • [SPARK-27991] [CORE] Defer the fetch request on Netty OOM
  • [SPARK-35171] [R] Declare the markdown package as a dependency of the SparkR package
  • [SPARK-35454] [SQL] One LogicalPlan can match multiple dataset ids
  • [SPARK-35298] [SQL] Migrate to transformWithPruning for rules in Optimizer.scala
  • [SPARK-35480] [SQL] Make percentile_approx work with pivot
  • [SPARK-35093] [SQL] AQE now uses newQueryStage plan as key for looking up cached exchanges for re-use
  • [SPARK-35146] [SQL] Migrate to transformWithPruning or resolveWithPruning for rules in finishAnalysis.scala
  • [SPARK-35411] [SQL] Add essential information while serializing TreeNode to json
  • [SPARK-35294] [SQL] Add tree traversal pruning in rules with dedicated files under optimizer
  • [SPARK-34897] [SQL][3.1] Support reconcile schemas based on index after nested column pruning
  • [SPARK-35144] [SQL] Migrate to transformWithPruning for object rules
  • [SPARK-35155] [SQL] Add rule id pruning to Analyzer rules
  • [SPARK-35382] [PYTHON] Fix lambda variable name issues in nested DataFrame functions in Python APIs.
  • [SPARK-35359] [SQL] Insert data with char/varchar datatype will fail when data length exceed length limitation
  • [SPARK-35381] [R] Fix lambda variable name issues in nested higher order functions at R APIs

System environment

  • Operating System: Ubuntu 18.04.5 LTS
  • Java: Zulu 8.54.0.21-CA-linux64
  • Scala: 2.12.10
  • Python: 3.8.8
  • R: 4.1.0 (2021-05-18)
  • Delta Lake 1.0.0

Installed Python libraries

Library Version Library Version Library Version
appdirs 1.4.4 asn1crypto 1.4.0 backcall 0.2.0
boto3 1.16.7 botocore 1.19.7 brotlipy 0.7.0
certifi 2021.5.30 cffi 1.14.3 chardet 3.0.4
cryptography 3.1.1 cycler 0.10.0 Cython 0.29.21
decorator 4.4.2 distlib 0.3.2 docutils 0.15.2
entrypoints 0.3 facets-overview 1.0.0 filelock 3.0.12
idna 2.10 ipykernel 5.3.4 ipython 7.19.0
ipython-genutils 0.2.0 jedi 0.17.2 jmespath 0.10.0
joblib 0.17.0 jupyter-client 6.1.7 jupyter-core 4.6.3
kiwisolver 1.3.0 koalas 1.8.1 matplotlib 3.2.2
numpy 1.19.2 pandas 1.1.5 parso 0.7.0
patsy 0.5.1 pexpect 4.8.0 pickleshare 0.7.5
pip 20.2.4 plotly 4.14.3 prompt-toolkit 3.0.8
protobuf 3.17.3 psycopg2 2.8.5 ptyprocess 0.6.0
pyarrow 1.0.1 pycparser 2.20 Pygments 2.7.2
pyOpenSSL 19.1.0 pyparsing 2.4.7 PySocks 1.7.1
python-dateutil 2.8.1 pytz 2020.5 pyzmq 19.0.2
requests 2.24.0 retrying 1.3.3 s3transfer 0.3.6
scikit-learn 0.23.2 scipy 1.5.2 seaborn 0.10.0
setuptools 50.3.1 six 1.15.0 statsmodels 0.12.0
threadpoolctl 2.1.0 tornado 6.0.4 traitlets 5.0.5
urllib3 1.25.11 virtualenv 20.2.1 wcwidth 0.2.5
wheel 0.35.1        

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.2.1
base 4.1.0 base64enc 0.1-3 BH 1.72.0-3
bit 4.0.4 bit64 4.0.5 blob 1.2.1
boot 1.3-28 brew 1.0-6 brio 1.1.0
broom 0.7.2 callr 3.5.1 caret 6.0-86
cellranger 1.1.0 chron 2.3-56 class 7.3-19
cli 2.2.0 clipr 0.7.1 cluster 2.1.2
codetools 0.2-18 colorspace 2.0-0 commonmark 1.7
compiler 4.1.0 config 0.3 covr 3.5.1
cpp11 0.2.4 crayon 1.3.4 credentials 1.3.0
crosstalk 1.1.0.1 curl 4.3 data.table 1.13.4
datasets 4.1.0 DBI 1.1.0 dbplyr 2.0.0
desc 1.2.0 devtools 2.3.2 diffobj 0.3.2
digest 0.6.27 dplyr 1.0.2 DT 0.16
ellipsis 0.3.1 evaluate 0.14 fansi 0.4.1
farver 2.0.3 fastmap 1.0.1 forcats 0.5.0
foreach 1.5.1 foreign 0.8-81 forge 0.2.0
fs 1.5.0 future 1.21.0 generics 0.1.0
gert 1.0.2 ggplot2 3.3.2 gh 1.2.0
gitcreds 0.1.1 glmnet 4.0-2 globals 0.14.0
glue 1.4.2 gower 0.2.2 graphics 4.1.0
grDevices 4.1.0 grid 4.1.0 gridExtra 2.3
gsubfn 0.7 gtable 0.3.0 haven 2.3.1
highr 0.8 hms 0.5.3 htmltools 0.5.0
htmlwidgets 1.5.3 httpuv 1.5.4 httr 1.4.2
hwriter 1.3.2 hwriterPlus 1.0-3 ini 0.3.1
ipred 0.9-9 isoband 0.2.3 iterators 1.0.13
jsonlite 1.7.2 KernSmooth 2.23-20 knitr 1.30
labeling 0.4.2 later 1.1.0.1 lattice 0.20-44
lava 1.6.8.1 lazyeval 0.2.2 lifecycle 0.2.0
listenv 0.8.0 lubridate 1.7.9.2 magrittr 2.0.1
markdown 1.1 MASS 7.3-54 Matrix 1.3-3
memoise 1.1.0 methods 4.1.0 mgcv 1.8-35
mime 0.9 ModelMetrics 1.2.2.2 modelr 0.1.8
munsell 0.5.0 nlme 3.1-152 nnet 7.3-16
numDeriv 2016.8-1.1 openssl 1.4.3 parallel 4.1.0
parallelly 1.22.0 pillar 1.4.7 pkgbuild 1.1.0
pkgconfig 2.0.3 pkgload 1.1.0 plogr 0.2.0
plyr 1.8.6 praise 1.0.0 prettyunits 1.1.1
pROC 1.16.2 processx 3.4.5 prodlim 2019.11.13
progress 1.2.2 promises 1.1.1 proto 1.0.0
ps 1.5.0 purrr 0.3.4 r2d3 0.2.3
R6 2.5.0 randomForest 4.6-14 rappdirs 0.3.1
rcmdcheck 1.3.3 RColorBrewer 1.1-2 Rcpp 1.0.5
readr 1.4.0 readxl 1.3.1 recipes 0.1.15
rematch 1.0.1 rematch2 2.1.2 remotes 2.2.0
reprex 0.3.0 reshape2 1.4.4 rex 1.2.0
rlang 0.4.9 rmarkdown 2.6 RODBC 1.3-17
roxygen2 7.1.1 rpart 4.1-15 rprojroot 2.0.2
Rserve 1.8-8 RSQLite 2.2.1 rstudioapi 0.13
rversions 2.0.2 rvest 0.3.6 scales 1.1.1
selectr 0.4-2 sessioninfo 1.1.1 shape 1.4.5
shiny 1.5.0 sourcetools 0.1.7 sparklyr 1.5.2
SparkR 3.1.1 spatial 7.3-11 splines 4.1.0
sqldf 0.4-11 SQUAREM 2020.5 stats 4.1.0
stats4 4.1.0 stringi 1.5.3 stringr 1.4.0
survival 3.2-11 sys 3.4 tcltk 4.1.0
TeachingDemos 2.10 testthat 3.0.0 tibble 3.0.4
tidyr 1.1.2 tidyselect 1.1.0 tidyverse 1.3.0
timeDate 3043.102 tinytex 0.28 tools 4.1.0
usethis 2.0.0 utf8 1.1.4 utils 4.1.0
uuid 0.1-4 vctrs 0.3.5 viridisLite 0.3.0
waldo 0.2.3 whisker 0.4 withr 2.3.0
xfun 0.19 xml2 1.3.2 xopen 1.0.0
xtable 1.8-4 yaml 2.2.1 zip 2.1.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.11.655
com.amazonaws aws-java-sdk-cloudformation 1.11.655
com.amazonaws aws-java-sdk-cloudfront 1.11.655
com.amazonaws aws-java-sdk-cloudhsm 1.11.655
com.amazonaws aws-java-sdk-cloudsearch 1.11.655
com.amazonaws aws-java-sdk-cloudtrail 1.11.655
com.amazonaws aws-java-sdk-cloudwatch 1.11.655
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.655
com.amazonaws aws-java-sdk-codedeploy 1.11.655
com.amazonaws aws-java-sdk-cognitoidentity 1.11.655
com.amazonaws aws-java-sdk-cognitosync 1.11.655
com.amazonaws aws-java-sdk-config 1.11.655
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-datapipeline 1.11.655
com.amazonaws aws-java-sdk-directconnect 1.11.655
com.amazonaws aws-java-sdk-directory 1.11.655
com.amazonaws aws-java-sdk-dynamodb 1.11.655
com.amazonaws aws-java-sdk-ec2 1.11.655
com.amazonaws aws-java-sdk-ecs 1.11.655
com.amazonaws aws-java-sdk-efs 1.11.655
com.amazonaws aws-java-sdk-elasticache 1.11.655
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.655
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.655
com.amazonaws aws-java-sdk-elastictranscoder 1.11.655
com.amazonaws aws-java-sdk-emr 1.11.655
com.amazonaws aws-java-sdk-glacier 1.11.655
com.amazonaws aws-java-sdk-glue 1.11.655
com.amazonaws aws-java-sdk-iam 1.11.655
com.amazonaws aws-java-sdk-importexport 1.11.655
com.amazonaws aws-java-sdk-kinesis 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-lambda 1.11.655
com.amazonaws aws-java-sdk-logs 1.11.655
com.amazonaws aws-java-sdk-machinelearning 1.11.655
com.amazonaws aws-java-sdk-marketplacemeteringservice 1.11.655
com.amazonaws aws-java-sdk-opsworks 1.11.655
com.amazonaws aws-java-sdk-rds 1.11.655
com.amazonaws aws-java-sdk-redshift 1.11.655
com.amazonaws aws-java-sdk-route53 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-ses 1.11.655
com.amazonaws aws-java-sdk-simpledb 1.11.655
com.amazonaws aws-java-sdk-simpleworkflow 1.11.655
com.amazonaws aws-java-sdk-sns 1.11.655
com.amazonaws aws-java-sdk-sqs 1.11.655
com.amazonaws aws-java-sdk-ssm 1.11.655
com.amazonaws aws-java-sdk-storagegateway 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws aws-java-sdk-support 1.11.655
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.655
com.amazonaws jmespath-java 1.11.655
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.fasterxml.jackson.datatype jackson-datatype-joda 2.10.0
com.fasterxml.jackson.module jackson-module-paranamer 2.10.0
com.fasterxml.jackson.module jackson-module-scala_2.12 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.joshelser dropwizard-metrics-hadoop-metrics2-reporter 0.1.2
com.github.luben zstd-jni 1.4.8-1
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.flatbuffers flatbuffers-java 1.9.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 1.4.195
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.9.5
com.twitter chill_2.12 0.9.5
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 3.1.0
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
hive-2.3__hadoop-2.7 jets3t-0.7 liball_deps_2.12
hive-2.3__hadoop-2.7 zookeeper-3.4 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.10
io.delta delta-sharing-spark_2.12 0.1.0
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.51.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.5
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
maven-trees hive-2.3__hadoop-2.7 liball_deps_2.12
net.java.dev.jna jna 5.8.0
net.razorvine pyrolite 4.30
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.3
net.snowflake spark-snowflake_2.12 2.9.0-spark_3.1
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8-1
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 2.0.0
org.apache.arrow arrow-memory-core 2.0.0
org.apache.arrow arrow-memory-netty 2.0.0
org.apache.arrow arrow-vector 2.0.0
org.apache.avro avro 1.8.2
org.apache.avro avro-ipc 1.8.2
org.apache.avro avro-mapred-hadoop2 1.8.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.10
org.apache.commons commons-math3 3.4.1
org.apache.commons commons-text 1.6
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.12.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-client 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.hadoop hadoop-hdfs 2.7.4
org.apache.hadoop hadoop-mapreduce-client-app 2.7.4
org.apache.hadoop hadoop-mapreduce-client-common 2.7.4
org.apache.hadoop hadoop-mapreduce-client-core 2.7.4
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.4
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.4
org.apache.hadoop hadoop-yarn-api 2.7.4
org.apache.hadoop hadoop-yarn-client 2.7.4
org.apache.hadoop hadoop-yarn-common 2.7.4
org.apache.hadoop hadoop-yarn-server-common 2.7.4
org.apache.hive hive-beeline 2.3.7
org.apache.hive hive-cli 2.3.7
org.apache.hive hive-jdbc 2.3.7
org.apache.hive hive-llap-client 2.3.7
org.apache.hive hive-llap-common 2.3.7
org.apache.hive hive-serde 2.3.7
org.apache.hive hive-shims 2.3.7
org.apache.hive hive-storage-api 2.7.2
org.apache.hive.shims hive-shims-0.23 2.3.7
org.apache.hive.shims hive-shims-common 2.3.7
org.apache.hive.shims hive-shims-scheduler 2.3.7
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.12
org.apache.ivy ivy 2.4.0
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.5.12
org.apache.orc orc-mapreduce 1.5.12
org.apache.orc orc-shims 1.5.12
org.apache.parquet parquet-column 1.10.1-databricks9
org.apache.parquet parquet-common 1.10.1-databricks9
org.apache.parquet parquet-encoding 1.10.1-databricks9
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1-databricks9
org.apache.parquet parquet-jackson 1.10.1-databricks9
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm7-shaded 4.15
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.4.14
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.36.v20210114
org.eclipse.jetty jetty-continuation 9.4.36.v20210114
org.eclipse.jetty jetty-http 9.4.36.v20210114
org.eclipse.jetty jetty-io 9.4.36.v20210114
org.eclipse.jetty jetty-jndi 9.4.36.v20210114
org.eclipse.jetty jetty-plus 9.4.36.v20210114
org.eclipse.jetty jetty-proxy 9.4.36.v20210114
org.eclipse.jetty jetty-security 9.4.36.v20210114
org.eclipse.jetty jetty-server 9.4.36.v20210114
org.eclipse.jetty jetty-servlet 9.4.36.v20210114
org.eclipse.jetty jetty-servlets 9.4.36.v20210114
org.eclipse.jetty jetty-util 9.4.36.v20210114
org.eclipse.jetty jetty-util-ajax 9.4.36.v20210114
org.eclipse.jetty jetty-webapp 9.4.36.v20210114
org.eclipse.jetty jetty-xml 9.4.36.v20210114
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.30
org.glassfish.jersey.containers jersey-container-servlet-core 2.30
org.glassfish.jersey.core jersey-client 2.30
org.glassfish.jersey.core jersey-common 2.30
org.glassfish.jersey.core jersey-server 2.30
org.glassfish.jersey.inject jersey-hk2 2.30
org.glassfish.jersey.media jersey-media-jaxb 2.30
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M5
org.json4s json4s-core_2.12 3.7.0-M5
org.json4s json4s-jackson_2.12 3.7.0-M5
org.json4s json4s-scalap_2.12 3.7.0-M5
org.lz4 lz4-java 1.7.1
org.mariadb.jdbc mariadb-java-client 2.2.5
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.1.4
org.roaringbitmap RoaringBitmap 0.9.14
org.roaringbitmap shims 0.9.14
org.rocksdb rocksdbjni 6.20.3
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.10
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-collection-compat_2.12 2.1.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.0
org.scalanlp breeze_2.12 1.0
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.30
org.slf4j jul-to-slf4j 1.7.30
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-log4j12 1.7.30
org.spark-project.spark unused 1.0.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.5
org.typelevel algebra_2.12 2.0.0-M2
org.typelevel cats-kernel_2.12 2.0.0-M4
org.typelevel machinist_2.12 0.6.8
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0-M1
org.typelevel spire-platform_2.12 0.17.0-M1
org.typelevel spire-util_2.12 0.17.0-M1
org.typelevel spire_2.12 0.17.0-M1
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.2
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52
antlr antlr 2.7.7
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws jmespath-java 1.11.655
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware.kryo kryo 2.21
com.esotericsoftware.minlog minlog 1.2
com.esotericsoftware.reflectasm reflectasm-shaded 1.07
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.googlecode.javaewah JavaEWAH 0.3.2
com.helger profiler 1.1.1
com.jolbox bonecp 0.8.0.RELEASE
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.thoughtworks.paranamer paranamer 2.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter parquet-hadoop-bundle 1.3.2
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-codec commons-codec 1.8
commons-collections commons-collections 3.2.2
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-io commons-io 2.5
commons-lang commons-lang 2.6
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
hive-2.3__hadoop-2.7 jets3t-0.7 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty 3.8.0.Final
io.netty netty-all 4.1.51.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
jakarta.validation jakarta.validation-api 2.0.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javolution javolution 5.5.1
jline jline 0.9.94
joda-time joda-time 2.10.5
junit junit 3.8.1
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
log4j log4j 1.2.17
net.sf.jpam jpam 1.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.4
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.avro avro 1.8.2
org.apache.avro avro 1.8.2
org.apache.commons commons-compress 1.20
org.apache.commons commons-compress 1.9
org.apache.commons commons-lang3 3.10
org.apache.commons commons-lang3 3.4
org.apache.commons commons-math3 3.4.1
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.10.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.4.1
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.1
org.apache.httpcomponents httpcore 4.4.12
org.apache.parquet parquet-column 1.10.1-databricks9
org.apache.parquet parquet-common 1.10.1-databricks9
org.apache.parquet parquet-encoding 1.10.1-databricks9
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1-databricks9
org.apache.parquet parquet-jackson 1.10.1-databricks9
org.apache.thrift libfb303 0.9.0
org.apache.thrift libthrift 0.9.2
org.apache.velocity velocity 1.5
org.apache.zookeeper zookeeper 3.4.6
org.codehaus.groovy groovy-all 2.1.6
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.datanucleus datanucleus-api-jdo 3.2.6
org.datanucleus datanucleus-core 3.2.10
org.datanucleus datanucleus-rdbms 3.2.9
org.eclipse.jetty jetty-client 9.4.36.v20210114
org.eclipse.jetty jetty-continuation 9.4.36.v20210114
org.eclipse.jetty jetty-http 9.4.36.v20210114
org.eclipse.jetty jetty-io 9.4.36.v20210114
org.eclipse.jetty jetty-proxy 9.4.36.v20210114
org.eclipse.jetty jetty-security 9.4.36.v20210114
org.eclipse.jetty jetty-server 9.4.36.v20210114
org.eclipse.jetty jetty-servlet 9.4.36.v20210114
org.eclipse.jetty jetty-servlets 9.4.36.v20210114
org.eclipse.jetty jetty-util 9.4.36.v20210114
org.eclipse.jetty jetty-util-ajax 9.4.36.v20210114
org.hibernate.validator hibernate-validator 6.1.0.Final
org.iq80.snappy snappy 0.2
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.json json 20090211
org.objenesis objenesis 1.2
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scalactic scalactic_2.12 3.0.8
org.scalatest scalatest_2.12 3.0.8
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-api 1.7.5
org.slf4j slf4j-log4j12 1.7.30
org.slf4j slf4j-log4j12 1.7.5
org.spark-project.hive hive-ant 0.13.1a
org.spark-project.hive hive-beeline 0.13.1a
org.spark-project.hive hive-cli 0.13.1a
org.spark-project.hive hive-common 0.13.1a
org.spark-project.hive hive-exec 0.13.1a
org.spark-project.hive hive-jdbc 0.13.1a
org.spark-project.hive hive-metastore 0.13.1a
org.spark-project.hive hive-serde 0.13.1a
org.spark-project.hive hive-service 0.13.1a
org.spark-project.hive hive-shims 0.13.1a
org.spark-project.hive.shims hive-shims-0.20 0.13.1a
org.spark-project.hive.shims hive-shims-0.20S 0.13.1a
org.spark-project.hive.shims hive-shims-0.23 0.13.1a
org.spark-project.hive.shims hive-shims-common 0.13.1a
org.spark-project.hive.shims hive-shims-common-secure 0.13.1a
org.spark-project.protobuf protobuf-java 2.5.0-spark
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.tukaani xz 1.5
org.tukaani xz 1.5
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial.snappy snappy-java 1.1.2.6
org.xerial.snappy snappy-java 1.1.8.2
oro oro 2.0.8
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52