Databricks Runtime 11.1

The following release notes provide information about Databricks Runtime 11.1, powered by Apache Spark 3.3.0. Databricks released these images in July 2022.

New features and improvements

Photon is GA

Photon is now generally available, beginning with Databricks Runtime 11.1. Photon is the native vectorized query engine on Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code. Photon is developed in C++ to take advantage of modern hardware, and uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications—all natively on your data lake.

Photon is part of a high-performance runtime that runs your existing SQL and DataFrame API calls faster and reduces your total cost per workload. Photon is used by default in Databricks SQL warehouses.

New features and improvements include:

  • New vectorized sort operator

  • New vectorized window functions

  • New instance types and sizes across all clouds

Limitations:

  • Scala/Python UDFs are not supported by Photon

  • RDD is not supported by Photon

  • Structured Streaming is not supported by Photon

For more information, see the following Photon announcements.

Photon: New vectorized sort operator

Photon now supports a vectorized sort for when a query contains SORT_BY, CLUSTER_BY, or a window function with an ORDER BY.

Limitations: Photon does not support a global ORDER BY clause. Sorts for window evaluation will photonize, but the global sort will continue to run in Spark.

Photon: New vectorized window functions

Photon now supports vectorized window function evaluation for many frame types and functions. New window functions include: row_number, rank, dense_rank, lag, lead, percent_rank, ntile, and nth_value. Supported window frame types: running (UNBOUNDED PRECEDING AND CURRENT ROW), unbounded (UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING), growing (UNBOUNDED PRECEDING AND <OFFSET> FOLLOWING), and shrinking (<OFFSET> PRECEDING AND UNBOUNDED FOLLOWING).

Limitations:

  • Photon supports only ROWS versions of all the frame types.

  • Photon does not yet support the sliding frame type (<OFFSET> PRECEDING AND <OFFSET> FOLLOWING).

Photon: Supported instance types

i3

i3en

i4i

im4gn

is4gen

m5ad

m5d

m5dn

m6gd

r5d

r5dn

r6gd

x2gd

Change data feed can now automatically handle out-of-range timestamps

Change data feed (CDF) now has a new mode for you to provide timestamps or versions past a latest commit’s version without throwing errors. This mode is disabled by default. You can enable it by setting the configuration spark.databricks.delta.changeDataFeed.timestampOutOfRange.enabled to true.

Describe and show SQL functions now show Unity Catalog names in their output (Public Preview)

The commands DESC TABLE, DESC DATABASE, DESC SCHEMA, DESC NAMESPACE, DESC FUNCTION, EXPLAIN, and SHOW CREATE TABLE now always show the catalog name in their output.

Schema inference and evolution for Parquet files in Auto Loader (Public Preview)

Auto Loader now supports schema inference and evolution for Parquet files. Just like JSON, CSV, and Avro formats, you can now use the rescued data column to rescue unexpected data that may appear in your Parquet files. This includes data that cannot be parsed in the data type that’s expected, columns that have a different casing, or additional columns that are not part of the expected schema. You can configure Auto Loader to evolve the schema automatically when encountering adding new columns in the incoming data. See Configuring schema inference and evolution in Auto Loader.

Delta Lake support for dynamic partition overwrites

Delta Lake now enables dynamic partition overwrite mode to overwrite all existing data in each logical partition for which the write will commit new data. See Dynamic Partition Overwrites.

Information schema support for objects created in Unity Catalog

Information schema provides a SQL based, self describing API to the metadata of various database objects, including tables and views, constraints and routines. Within the information schema you find a set of views describing the objects known to the schema’s catalog that you are privileged the see. The information schema of the SYSTEM catalog returns information about objects across all catalogs within the metastore. See Information schema.

Informational constraints on Delta Lake tables with Unity Catalog (Public Preview)

You can now define informational primary key and foreign key constraints on Delta Lake tables with Unity Catalog. Informational constraints are not enforced. See CONSTRAINT clause (Databricks Runtime).

Behavior changes

Sensitive properties redaction for DESCRIBE TABLE and SHOW TABLE PROPERTIES

The DESCRIBE TABLE and SHOW TABLE PROPERTIES commands now redact sensitive properties.

Library upgrades

  • Upgraded Python libraries:

    • filelock from 3.6.0 to 3.7.1

    • plotly from 5.6.0 to 5.8.2

    • protobuf from 3.20.1 to 4.21.2

  • Upgraded R libraries:

    • chron from 2.3-56 to 2.3-57

    • DBI from 1.1.2 to 1.1.3

    • dbplyr from 2.1.1 to 2.2.0

    • e1071 from 1.7-9 to 1.7-11

    • future from 1.25.0 to 1.26.1

    • globals from 0.14.0 to 0.15.1

    • hardhat from 0.2.0 to 1.1.0

    • ipred from 0.9-12 to 0.9-13

    • openssl from 2.0.0 to 2.0.2

    • parallelly from 1.31.1 to 1.32.0

    • processx from 3.5.3 to 3.6.1

    • progressr from 0.10.0 to 0.10.1

    • proxy from 0.4-26 to 0.4-27

    • ps from 1.7.0 to 1.7.1

    • randomForest from 4.7-1 to 4.7-1.1

    • roxygen2 from 7.1.2 to 7.2.0

    • Rserve from 1.8-10 to 1.8-11

    • RSQLite from 2.2.13 to 2.2.14

    • sparklyr from 1.7.5 to 1.7.7

    • tinytex from 0.38 to 0.40

    • usethis from 2.1.5 to 2.1.6

    • xfun from 0.30 to 0.31

  • Upgraded Java libraries:

    • io.delta.delta-sharing-spark_2.12 from 0.4.0 to 0.5.0

Apache Spark

Databricks Runtime 11.1 includes Apache Spark 3.3.0. This release includes all Spark fixes and improvements included in Databricks Runtime 11.0, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-39449] [SQL] Propagate empty relation through Window

  • [SPARK-39570] [WARMFIX][SQL] Inline table should allow expressions with alias

  • [SPARK-38796] [WARMFIX][SQL] Update to_number and try_to_number functions to allow PR with positive numbers

  • [SPARK-39649] [PYTHON] Make listDatabases / getDatabase / listColumns / refreshTable in PySpark support 3-layer-namespace

  • [SPARK-39646] [SQL] Make setCurrentDatabase compatible with 3 layer namespace

  • [SPARK-39645] [SQL] Make getDatabase and listDatabases compatible with 3 layer namespace

  • [SPARK-39583] [SQL] Make RefreshTable be compatible with 3 layer namespace

  • [SPARK-39598] [PYTHON] Make cache, catalog in the python side support 3-layer-namespace

  • [SPARK-39597] [PYTHON] Make GetTable, TableExists and DatabaseExists in the python side support 3-layer-namespace

  • [SPARK-39615] [SQL] Make listColumns be compatible with 3 layer namespace

  • [SPARK-39555] [PYTHON] Make createTable and listTables in the python side support 3-layer-namespace

  • [SPARK-39506] [SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible

  • [SPARK-39263] [SQL] Make GetTable, TableExists and DatabaseExists be compatible with 3 layer namespace

  • [SPARK-39236] [SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

  • [SPARK-39451] [SQL] Support casting intervals to integrals in ANSI mode

  • [SPARK-39346] [SQL][3.3] Convert asserts/illegal state exception to internal errors on each phase

  • [SPARK-39272] [SQL] Increase the start position of query context by 1

  • [SPARK-39392] [SQL] Refine ANSI error messages for try_* function hints

  • [SPARK-39413] [SQL] Capitalize sql keywords in JDBCV2Suite

  • [SPARK-39393] [SQL] Parquet data source only supports push-down predicate filters for non-repeated primitive types

  • [SPARK-38289] [SQL] Refactor SQL CLI exit code to make it more clear

  • [SPARK-39404] [SS] Minor fix for querying _metadata in streaming

  • [SPARK-39417] [SQL] Handle Null partition values in PartitioningUtils

  • [SPARK-39359] [SQL] Restrict DEFAULT columns to allowlist of supported data source types

  • [SPARK-39294] [SQL] Support vectorized Orc scans with DEFAULT values

  • [SPARK-39422] [SQL] Improve error message for ‘SHOW CREATE TABLE’ with unsupported serdes

  • [SPARK-39265] [SQL] Support vectorized Parquet scans with DEFAULT values

  • [SPARK-39306] [SQL] Support scalar subquery in time travel

  • [SPARK-39368] [SQL] Move RewritePredicateSubquery into InjectRuntimeFilter

  • [SPARK-39167] [SQL] Throw an exception w/ an error class for multiple rows from a subquery used as an expression

  • [SPARK-39248] [SQL] Improve divide performance for decimal type

  • [SPARK-39266] [CORE] Cleanup unused spark.rpc.numRetries and spark.rpc.retry.wait configs

  • [SPARK-38807] [CORE] Fix the startup error of spark shell on Windows

  • [SPARK-38869] [Contd] Hide the new method from Delta Lake OSS export

  • [SPARK-39338] [SQL] Remove dynamic pruning subquery if pruningKey’s references is empty

  • [SPARK-39265] [SQL] Support non-vectorized Parquet scans with DEFAULT values

  • [SPARK-39350] [SQL] DESC NAMESPACE EXTENDED should redact properties

  • [SPARK-39230] [SQL] Support ANSI Aggregate Function: regr_slope

  • [SPARK-39277] [SQL] Make Optimizer extends SQLConfHelper and remove incorrect comment

  • [SPARK-39296] [CORE][SQL] Replcace Array.toString with Array.mkString

  • [SPARK-39333] [SQL] Change to use foreach when map doesn’t produce results

  • [SPARK-39294] [SQL] Support non-vectorized Orc scans with DEFAULT values

  • [SPARK-38687] [SQL] Use error classes in the compilation errors of generators

  • [SPARK-39234] [SQL] Code clean up in SparkThrowableHelper.getMessage

  • [SPARK-39163] [SQL] Throw an exception w/ error class for an invalid bucket file

  • [SPARK-39229] [SQL] Separate query contexts from error-classes.json

  • [SPARK-39264] [SS] Fix type check and conversion to longOffset for awaitOffset fix

  • [SPARK-37523] [SQL] Re-optimize partitions in Distribution and Ordering if numPartitions is not specified

  • [SPARK-39270] [SQL] JDBC dialect supports registering dialect specific functions

  • [SPARK-39305] [SQL] Add the EQUAL_NULL() function

  • [SPARK-39212] [SQL] Use double quotes for values of SQL configs/DS options in error messages

  • [SPARK-38869] [SQL] Respect table capability ACCEPT_ANY_SCHEMA in DEFAULT column resolution

  • [SPARK-39268] [SQL] AttachDistributedSequenceExec do not checkpoint childRDD with single partition

  • [SPARK-39293] [SQL] Fix the accumulator of ArrayAggregate to handle complex types properly

  • [SPARK-39211] [SQL] Support JSON scans with DEFAULT values

  • [SPARK-39216] [SQL] Do not collapse projects in CombineUnions if it hasCorrelatedSubquery

  • [SPARK-39214] [SQL][3.3] Improve errors related to CAST

  • [SPARK-38897] [SQL] DS V2 supports push down string functions

  • [SPARK-39260] [SQL] Use Reader.getSchema instead of Reader.getTypes in SparkOrcNewRecordReader

  • [SPARK-39242] [SS] Fix awaitOffset to wait for committedOffset to reach at-least expected offset for longOffset and fix RateStreamProvider test

  • [SPARK-37915] [SQL] Combine unions if there is a project between them

  • [SPARK-38681] [SQL] Support nested generic case classes

  • [SPARK-39143] [SQL] Support CSV scans with DEFAULT values

  • [SPARK-28516] [SQL] Implement to_char and try_to_char functions to format Decimal values as strings

  • [SPARK-39081] [PYTHON][SQL] Implement DataFrame.resample and Series.resample

  • [SPARK-39106] [SQL] Correct conditional expression constant folding

  • [SPARK-28516] [UC] Add override

  • [SPARK-39104] [SQL] InMemoryRelation#isCachedColumnBuffersLoaded should be thread-safe

System environment

  • Operating System: Ubuntu 20.04.4 LTS

  • Java: Zulu 8.56.0.21-CA-linux64

  • Scala: 2.12.14

  • Python: 3.9.5

  • R: 4.1.3

  • Delta Lake: 2.0.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

Antergos Linux

2015.10 (ISO-Rolling)

argon2-cffi

20.1.0

async-generator

1.10

attrs

21.2.0

backcall

0.2.0

backports.entry-points-selectable

1.1.1

bleach

4.0.0

boto3

1.21.18

botocore

1.24.18

certifi

2021.10.8

cffi

1.14.6

chardet

4.0.0

charset-normalizer

2.0.4

cycler

0.10.0

Cython

0.29.24

dbus-python

1.2.16

debugpy

1.4.1

decorator

5.1.0

defusedxml

0.7.1

distlib

0.3.4

distro-info

0.23ubuntu1

entrypoints

0.3

facets-overview

1.0.0

filelock

3.7.1

idna

3.2

ipykernel

6.12.1

ipython

7.32.0

ipython-genutils

0.2.0

ipywidgets

7.7.0

jedi

0.18.0

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.0.1

jsonschema

3.2.0

jupyter-client

6.1.12

jupyter-core

4.8.1

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.1

MarkupSafe

2.0.1

matplotlib

3.4.3

matplotlib-inline

0.1.2

mistune

0.8.4

nbclient

0.5.3

nbconvert

6.1.0

nbformat

5.1.3

nest-asyncio

1.5.1

notebook

6.4.5

numpy

1.20.3

packaging

21.0

pandas

1.3.4

pandocfilters

1.4.3

parso

0.8.2

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

8.4.0

pip

21.2.4

platformdirs

2.5.2

plotly

5.8.2

prometheus-client

0.11.0

prompt-toolkit

3.0.20

protobuf

4.21.2

psutil

5.8.0

psycopg2

2.9.3

ptyprocess

0.7.0

pyarrow

7.0.0

pycparser

2.20

Pygments

2.10.0

PyGObject

3.36.0

pyodbc

4.0.31

pyparsing

3.0.4

pyrsistent

0.18.0

python-apt

2.0.0+ubuntu0.20.4.7

python-dateutil

2.8.2

pytz

2021.3

pyzmq

22.2.1

requests

2.26.0

requests-unixsocket

0.2.0

s3transfer

0.5.2

scikit-learn

0.24.2

scipy

1.7.1

seaborn

0.11.2

Send2Trash

1.8.0

setuptools

58.0.4

six

1.16.0

ssh-import-id

5.10

statsmodels

0.12.2

tenacity

8.0.1

terminado

0.9.4

testpath

0.5.0

threadpoolctl

2.2.0

tornado

6.1

traitlets

5.1.0

unattended-upgrades

0.1

urllib3

1.26.7

virtualenv

20.8.0

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.37.0

widgetsnbextension

3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-06-26.

Library

Version

Library

Version

Library

Version

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.1.3

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-7

brio

1.1.3

broom

0.8.0

bslib

0.3.1

cachem

1.0.6

callr

3.7.0

caret

6.0-92

cellranger

1.1.0

chron

2.3-57

class

7.3-20

cli

3.3.0

clipr

0.8.0

cluster

2.1.3

codetools

0.2-18

colorspace

2.0-3

commonmark

1.8.0

compiler

4.1.3

config

0.3.1

cpp11

0.4.2

crayon

1.5.1

credentials

1.3.2

curl

4.3.2

data.table

1.14.2

datasets

4.1.3

DBI

1.1.3

dbplyr

2.2.0

desc

1.4.1

devtools

2.4.3

diffobj

0.3.5

digest

0.6.29

dplyr

1.0.9

dtplyr

1.2.1

e1071

1.7-11

ellipsis

0.3.2

evaluate

0.15

fansi

1.0.3

farver

2.1.0

fastmap

1.1.0

fontawesome

0.2.2

forcats

0.5.1

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.26.1

future.apply

1.9.0

gargle

1.2.0

generics

0.1.2

gert

1.6.0

ggplot2

3.3.6

gh

1.3.0

gitcreds

0.1.1

glmnet

4.1-4

globals

0.15.1

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.0

gower

1.0.0

graphics

4.1.3

grDevices

4.1.3

grid

4.1.3

gridExtra

2.3

gsubfn

0.7

gtable

0.3.0

hardhat

1.1.0

haven

2.5.0

highr

0.9

hms

1.1.1

htmltools

0.5.2

htmlwidgets

1.5.4

httpuv

1.6.5

httr

1.4.3

ids

1.0.1

ini

0.3.1

ipred

0.9-13

isoband

0.2.5

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.0

KernSmooth

2.23-20

knitr

1.39

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.6.10

lifecycle

1.0.1

listenv

0.8.0

lubridate

1.8.0

magrittr

2.0.3

markdown

1.1

MASS

7.3-56

Matrix

1.4-1

memoise

2.0.1

methods

4.1.3

mgcv

1.8-40

mime

0.12

ModelMetrics

1.2.2.2

modelr

0.1.8

munsell

0.5.0

nlme

3.1-157

nnet

7.3-17

numDeriv

2016.8-1.1

openssl

2.0.2

parallel

4.1.3

parallelly

1.32.0

pillar

1.7.0

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgload

1.2.4

plogr

0.2.0

plyr

1.8.7

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.6.1

prodlim

2019.11.13

progress

1.2.2

progressr

0.10.1

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.1

purrr

0.3.4

r2d3

0.2.6

R6

2.5.1

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.8.3

RcppEigen

0.3.3.9.2

readr

2.1.2

readxl

1.4.0

recipes

0.2.0

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.1

reshape2

1.4.4

rlang

1.0.2

rmarkdown

2.14

RODBC

1.3-19

roxygen2

7.2.0

rpart

4.1.16

rprojroot

2.0.3

Rserve

1.8-11

RSQLite

2.2.14

rstudioapi

0.13

rversions

2.1.1

rvest

1.0.2

sass

0.4.1

scales

1.2.0

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.1

sourcetools

0.1.7

sparklyr

1.7.7

SparkR

3.3.0

spatial

7.3-11

splines

4.1.3

sqldf

0.4-11

SQUAREM

2021.1

stats

4.1.3

stats4

4.1.3

stringi

1.7.6

stringr

1.4.0

survival

3.2-13

sys

3.4

tcltk

4.1.3

testthat

3.1.4

tibble

3.1.7

tidyr

1.2.0

tidyselect

1.1.2

tidyverse

1.3.1

timeDate

3043.102

tinytex

0.40

tools

4.1.3

tzdb

0.3.0

usethis

2.1.6

utf8

1.2.2

utils

4.1.3

uuid

1.1-0

vctrs

0.4.1

viridisLite

0.4.0

vroom

1.5.7

waldo

0.4.0

whisker

0.4

withr

2.5.0

xfun

0.31

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.5

zip

2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.189

com.amazonaws

aws-java-sdk-cloudformation

1.12.189

com.amazonaws

aws-java-sdk-cloudfront

1.12.189

com.amazonaws

aws-java-sdk-cloudhsm

1.12.189

com.amazonaws

aws-java-sdk-cloudsearch

1.12.189

com.amazonaws

aws-java-sdk-cloudtrail

1.12.189

com.amazonaws

aws-java-sdk-cloudwatch

1.12.189

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.189

com.amazonaws

aws-java-sdk-codedeploy

1.12.189

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.189

com.amazonaws

aws-java-sdk-cognitosync

1.12.189

com.amazonaws

aws-java-sdk-config

1.12.189

com.amazonaws

aws-java-sdk-core

1.12.189

com.amazonaws

aws-java-sdk-datapipeline

1.12.189

com.amazonaws

aws-java-sdk-directconnect

1.12.189

com.amazonaws

aws-java-sdk-directory

1.12.189

com.amazonaws

aws-java-sdk-dynamodb

1.12.189

com.amazonaws

aws-java-sdk-ec2

1.12.189

com.amazonaws

aws-java-sdk-ecs

1.12.189

com.amazonaws

aws-java-sdk-efs

1.12.189

com.amazonaws

aws-java-sdk-elasticache

1.12.189

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.189

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.189

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.189

com.amazonaws

aws-java-sdk-emr

1.12.189

com.amazonaws

aws-java-sdk-glacier

1.12.189

com.amazonaws

aws-java-sdk-glue

1.12.189

com.amazonaws

aws-java-sdk-iam

1.12.189

com.amazonaws

aws-java-sdk-importexport

1.12.189

com.amazonaws

aws-java-sdk-kinesis

1.12.189

com.amazonaws

aws-java-sdk-kms

1.12.189

com.amazonaws

aws-java-sdk-lambda

1.12.189

com.amazonaws

aws-java-sdk-logs

1.12.189

com.amazonaws

aws-java-sdk-machinelearning

1.12.189

com.amazonaws

aws-java-sdk-opsworks

1.12.189

com.amazonaws

aws-java-sdk-rds

1.12.189

com.amazonaws

aws-java-sdk-redshift

1.12.189

com.amazonaws

aws-java-sdk-route53

1.12.189

com.amazonaws

aws-java-sdk-s3

1.12.189

com.amazonaws

aws-java-sdk-ses

1.12.189

com.amazonaws

aws-java-sdk-simpledb

1.12.189

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.189

com.amazonaws

aws-java-sdk-sns

1.12.189

com.amazonaws

aws-java-sdk-sqs

1.12.189

com.amazonaws

aws-java-sdk-ssm

1.12.189

com.amazonaws

aws-java-sdk-storagegateway

1.12.189

com.amazonaws

aws-java-sdk-sts

1.12.189

com.amazonaws

aws-java-sdk-support

1.12.189

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.189

com.amazonaws

jmespath-java

1.12.189

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.13.3

com.fasterxml.jackson.core

jackson-core

2.13.3

com.fasterxml.jackson.core

jackson-databind

2.13.3

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.13.3

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.13.3

com.fasterxml.jackson.module

jackson-module-paranamer

2.13.3

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.13.3

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.2-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.1

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.0.204

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.ning

compress-lzf

1.1

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

hadoop3

jets3t-0.7

liball_deps_2.12

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.5.0

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.74.Final

io.netty

netty-buffer

4.1.74.Final

io.netty

netty-codec

4.1.74.Final

io.netty

netty-common

4.1.74.Final

io.netty

netty-handler

4.1.74.Final

io.netty

netty-resolver

4.1.74.Final

io.netty

netty-tcnative-classes

2.0.48.Final

io.netty

netty-transport

4.1.74.Final

io.netty

netty-transport-classes-epoll

4.1.74.Final

io.netty

netty-transport-classes-kqueue

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-aarch_64

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-x86_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-aarch_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-x86_64

4.1.74.Final

io.netty

netty-transport-native-unix-common

4.1.74.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.13

mvn

hadoop3

liball_deps_2.12

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.2

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.14

net.snowflake

spark-snowflake_2.12

2.10.0-spark_3.2

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

7.0.0

org.apache.arrow

arrow-memory-core

7.0.0

org.apache.arrow

arrow-memory-netty

7.0.0

org.apache.arrow

arrow-vector

7.0.0

org.apache.avro

avro

1.11.0

org.apache.avro

avro-ipc

1.11.0

org.apache.avro

avro-mapred

1.11.0

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.9

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.2-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.2

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.14

org.apache.ivy

ivy

2.5.0

org.apache.logging.log4j

log4j-1.2-api

2.17.2

org.apache.logging.log4j

log4j-api

2.17.2

org.apache.logging.log4j

log4j-core

2.17.2

org.apache.logging.log4j

log4j-slf4j-impl

2.17.2

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.7.4

org.apache.orc

orc-mapreduce

1.7.4

org.apache.orc

orc-shims

1.7.4

org.apache.parquet

parquet-column

1.12.0-databricks-0004

org.apache.parquet

parquet-common

1.12.0-databricks-0004

org.apache.parquet

parquet-encoding

1.12.0-databricks-0004

org.apache.parquet

parquet-format-structures

1.12.0-databricks-0004

org.apache.parquet

parquet-hadoop

1.12.0-databricks-0004

org.apache.parquet

parquet-jackson

1.12.0-databricks-0004

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.5.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.46.v20220331

org.eclipse.jetty

jetty-continuation

9.4.46.v20220331

org.eclipse.jetty

jetty-http

9.4.46.v20220331

org.eclipse.jetty

jetty-io

9.4.46.v20220331

org.eclipse.jetty

jetty-jndi

9.4.46.v20220331

org.eclipse.jetty

jetty-plus

9.4.46.v20220331

org.eclipse.jetty

jetty-proxy

9.4.46.v20220331

org.eclipse.jetty

jetty-security

9.4.46.v20220331

org.eclipse.jetty

jetty-server

9.4.46.v20220331

org.eclipse.jetty

jetty-servlet

9.4.46.v20220331

org.eclipse.jetty

jetty-servlets

9.4.46.v20220331

org.eclipse.jetty

jetty-util

9.4.46.v20220331

org.eclipse.jetty

jetty-util-ajax

9.4.46.v20220331

org.eclipse.jetty

jetty-webapp

9.4.46.v20220331

org.eclipse.jetty

jetty-xml

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-api

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-client

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-common

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-server

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-servlet

9.4.46.v20220331

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.34

org.glassfish.jersey.containers

jersey-container-servlet-core

2.34

org.glassfish.jersey.core

jersey-client

2.34

org.glassfish.jersey.core

jersey-common

2.34

org.glassfish.jersey.core

jersey-server

2.34

org.glassfish.jersey.inject

jersey-hk2

2.34

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.3

org.roaringbitmap

RoaringBitmap

0.9.25

org.roaringbitmap

shims

0.9.25

org.rocksdb

rocksdbjni

6.24.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.36

org.slf4j

jul-to-slf4j

1.7.36

org.slf4j

slf4j-api

1.7.36

org.spark-project.spark

unused

1.0.0

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1