Databricks Runtime 10.2 (unsupported)

The following release notes provide information about Databricks Runtime 10.2 and Databricks Runtime 10.2 Photon, powered by Apache Spark 3.2.0. Databricks released these images in December 2021. Photon is in Public Preview.

New features and improvements

Use Files in Repos with Spark Streaming

You can now access Files in Repos from Spark Streaming.

Databricks Utilities adds an update mount command

The new dbutils.fs.updateMount command is similar to dbutils.fs.mount, but this new command updates an existing mount point instead of creating a new one. See updateMount command (dbutils.fs.updateMount).

Delta Lake supports renaming of columns (Public Preview)

Delta Lake now supports column renaming. To rename the column of a Delta table, you must upgrade your Delta table’s version and then set the column mapping mode to name mapping. See Rename and drop columns with Delta Lake column mapping.

Delta Lake table history adds Databricks Runtime version information

When a write is now made to a Delta table, information about the Databricks Runtime version that made the write is also recorded. To get the version information, run the DESCRIBE HISTORY command on the table and then see the engineInfo column in the output. See DESCRIBE HISTORY and Work with Delta Lake table history.

Databricks Utilities adds latest modification time information when directories are listed

The dbutils.fs.ls command for listing the contents of a directory now outputs a modificationTime field. This field contains the latest modification time for each listed file or directory. See ls command (dbutils.fs.ls).

Spark SQL adds an approx_top_k function to return top items and their counts

The new approx_top_k function returns the most frequently occurring item values, along with their approximate counts, for the specified expression. Some examples include:

df = spark.sql('SELECT approx_top_k(col, 10, 100) FROM VALUES (0), (1), (1), (2), (2), (2) AS tab(col)')

display(df)

import pyspark.sql.functions as F

df = spark.createDataFrame([
       (0,), (1,), (1,), (2,), (2,), (2,)
     ]).select(F.expr('approx_top_k(_1, 10)'))

display(df)

The result of running either example is:

[{"item": 2, "count": 3}, {"item": 1, "count": 2}, {"item": 0, "count": 1}]

See approx_top_k aggregate function.

Trigger.AvailableNow now available in Python as well as Scala

This change supports the Delta source streaming queries Trigger.AvailableNow in Python as well as Scala. Databricks Runtime 10.1 introduced support for this trigger in Scala. Databricks Runtime 10.2 now supports this trigger in Python as well. See coverage of Trigger.AvailableNow in What is Auto Loader?.

Auto Loader now supports schema evolution for Avro (Public Preview)

See Configure schema inference and evolution in Auto Loader.

Library upgrades

• Upgraded Python libraries:

  • filelock from 3.3.1 to 3.4.0

  • plotly from 5.3.0 to 5.3.1

• Upgraded R libraries:

  • base from 4.1.1 to 4.1.2

  • compiler from 4.1.1 to 4.1.2

  • datasets from 4.1.1 to 4.1.2

  • graphics from 4.1.1 to 4.1.2

  • grDevices from 4.1.1 to 4.1.2

  • grid from 4.1.1 to 4.1.2

  • methods from 4.1.1 to 4.1.2

  • parallel from 4.1.1 to 4.1.2

  • Rserve from 1.8-8 to 1.8-10

  • splines from 4.1.1 to 4.1.2

  • stats from 4.1.1 to 4.1.2

  • stats4 from 4.1.1 to 4.1.2

  • tcltk from 4.1.1 to 4.1.2

  • tools from 4.1.1 to 4.1.2

  • utils from 4.1.1 to 4.1.2

  • backports from 1.2.1 to 1.4.0

  • broom from 0.7.9 to 0.7.10

  • cli from 3.0.1 to 3.1.0

  • cpp11 from 0.4.0 to 0.4.1

  • crayon from 1.4.1 to 1.4.2

  • credentials from 1.3.1 to 1.3.2

  • future from 1.22.1 to 1.23.0

  • gert from 1.4.1 to 1.4.3

  • glmnet from 4.1-2 to 4.1-3

  • glue from 1.4.2 to 1.5.0

  • memoise from 2.0.0 to 2.0.1

  • parallelly from 1.28.1 to 1.29.0

  • readr from 2.0.2 to 2.1.0

  • sessioninfo from 1.1.1 to 1.2.1

  • stringi from 1.7.5 to 1.7.6

  • tibble from 3.1.5 to 3.1.6

  • tinytex from 0.34 to 0.35

  • tzdb from 0.1.2 to 0.2.0

  • usethis from 2.1.2 to 2.1.3

  • uuid from 0.1-4 to 1.0-3

  • vroom from 1.5.5 to 1.5.6

  • xfun from 0.27 to 0.28

• Upgraded Java libraries:

  • com.fasterxml.jackson from 2.12.2 to 2.12.3

  • io.airlift.aircompressor from 0.19 to 0.21

  • io.netty.netty-all from 4.1.63.Final to 4.1.68.Final

  • joda-time.joda-time from 2.10.5 to 2.10.10

  • org.apache.commons.commons-compress from 1.20 to 1.21

  • org.apache.orc from 1.6.10 to 1.6.12

  • org.eclipse.jetty from 9.4.40.v20210413 to 9.4.43.v20210629

  • org.scalanlp.breeze-macros_2.12 from 1.0 to 1.2

  • org.scalanlp.breeze_2.12 from 1.0 to 1.2

  • org.org.typelevel.algebra_2.12 from 2.0.0 to 2.0.1

  • org.typelevel.cats-kernel_2.12 from 2.0.0 to 2.0.1

Apache Spark

Databricks Runtime 10.2 includes Apache Spark 3.2.0. This release includes all Spark fixes and improvements included in Databricks Runtime 10.1 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-37447] [SQL] Cache LogicalPlan.isStreaming() result in a lazy val

• [SPARK-37468] [SQL] Support ANSI intervals and TimestampNTZ for UnionEstimation

• [SPARK-37464] [SQL] SCHEMA and DATABASE should simply be aliases of NAMESPACE

• [SPARK-37452] [SQL] Char and Varchar break backward compatibility between v3.1 and v2

• [SPARK-36861] [SQL] Use yyyy-MM-dd as the date pattern in partition discovery

• [SPARK-37456] [SQL] CREATE NAMESPACE should qualify location for v2 command

• [SPARK-37357] [SQL] Add small partition factor for rebalance partitions

• [SPARK-37446] [SQL] Use reflection for getWithoutRegisterFns to allow different Hive versions for building

• [SPARK-37444] [SQL] ALTER NAMESPACE … SET LOCATION should handle empty location consistently across v1 and v2 command

• [SPARK-36357] [SQL] Support pushdown Timestamp without time zone for orc

• [SPARK-36346] [SQL] Support TimestampNTZ type in Orc file source

• [SPARK-36646] [SQL] Push down group by partition column for aggregate

• [SPARK-34960] [SQL] Aggregate push down for ORC

• [SPARK-37386] [SQL] Simplify OptimizeSkewedJoin to not run the cost evaluator

• [SPARK-36073] [SQL] EquivalentExpressions fixes and improvements

• [SPARK-35352] [SQL] Add code-gen for full outer sort merge join

• [SPARK-37292] [SQL] Removes outer join if it only has DISTINCT on streamed side with alias

• [SPARK-37371] [SQL] UnionExec should support columnar if all children support columnar

• [SPARK-36645] [SQL] Aggregate (Min/Max/Count) push down for Parquet

• [SPARK-37379] [SQL] Add tree pattern pruning to CTESubstitution rule

• [SPARK-37332] [SQL] Allow ANSI intervals in ALTER TABLE .. ADD COLUMNS

• [SPARK-37224] [SS] Optimize write path on RocksDB state store provider

• [SPARK-37304] [SQL] Allow ANSI intervals in v2 ALTER TABLE .. REPLACE COLUMNS

• [SPARK-36924] [SQL] CAST between ANSI intervals and IntegralType

• [SPARK-37289] [SQL] Remove partitionSchemaOption function

• [SPARK-37270] [SQL] Fix push foldable into CaseWhen branches if elseValue is empty

• [SPARK-35345] [SQL] Add Parquet tests to BloomFilterBenchmark

• [SPARK-37341] [SQL] Avoid unnecessary buffer and copy in full outer sort merge join

• [SPARK-37266] [SQL] View text can only be SELECT queries

• [SPARK-37288] [PYTHON][3.2] Backport since annotation update

• [SPARK-37262] [SQL] Don’t log empty aggregate and group by in JDBCScan

• [SPARK-37283] [SQL] Don’t try to store a V1 table which contains ANSI intervals in Hive compatible format

• [SPARK-37221] [SQL] The collect-like API in SparkPlan should support columnar output

• [SPARK-37202] [SQL] Use AnalysisContext to track referred temp functions

• [SPARK-37023] [CORE] Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

• [SPARK-37307] [CORE] Don’t obtain JDBC connection for empty partitions

• [SPARK-37267] [SQL] OptimizeSkewInRebalancePartitions support optimize non-root node

• [SPARK-36799] [SQL] Pass queryExecution name in CLI

• [SPARK-37261] [SQL] Allow adding partitions with ANSI intervals in DSv2

• [SPARK-37203] [SQL] Fix NotSerializableException when observe with TypedImperativeAggregate

• [SPARK-37199] [SQL] Add deterministic field to QueryPlan

• [SPARK-37253] [PYTHON] try_simplify_traceback should not fail when tb_frame.f_lineno is None

• [SPARK-37214] [SQL] Fail query analysis earlier with invalid identifiers

• [SPARK-37240] [SQL] Handle ANSI intervals by ColumnVectorUtils.populate()

• [SPARK-35437] [SQL] Use expressions to filter Hive partitions at client side

• [SPARK-32940] [SQL] Collect, first and last should be deterministic aggregate functions

• [SPARK-37231] [SQL] Dynamic writes/reads of ANSI interval partitions

• [SPARK-37038] [SQL] DSV2 Sample Push Down

• [SPARK-36665] [SQL] Add more Not operator simplifications

• [SPARK-37150] [SQL] Migrate DESCRIBE NAMESPACE to use V2 command by default

• [SPARK-37196] [SQL] HiveDecimal enforcePrecisionScale failed return null

• [SPARK-37212] [SQL] Improve the implement of aggregate pushdown.

• [SPARK-37190] [SQL] Improve error messages for Cast under ANSI mode

• [SPARK-37161] [SQL] RowToColumnConverter support AnsiIntervalType

• [SPARK-37160] [SQL] Add a config to optionally disable padding for char type

• [SPARK-36554] [SQL][PYTHON] Expose make_date expression in functions.scala

• [SPARK-37200] [SQL] Support drop index for Data Source V2

• [SPARK-37179] [SQL] ANSI mode: Add a config to allow casting between Datetime and Numeric

• [SPARK-37168] [SQL] Improve error messages for SQL functions and operators under ANSI mode

• [SPARK-36935] [SQL] Extend ParquetSchemaConverter to compute Parquet repetition & definition level

• [SPARK-37066] [SQL] Improve error message to show file path when failed to read next file

• [SPARK-32567] [SQL] Add code-gen for full outer shuffled hash join

• [SPARK-37176] [SQL] Sync JsonInferSchema#infer method’s exception handle logic with JacksonParser#parse method

• [SPARK-37149] [SQL] Improve error messages for arithmetic overflow exceptions under ANSI mode

• [SPARK-37136] [SQL] Remove code about hive buildin function but not implement in spark

• [SPARK-16280] [SPARK-37082] [SQL] Implements histogram_numeric aggregation function which supports partial aggregation.

• [SPARK-36975] [SQL] Correct the hive client calls‘s metrics in HiveClientImpl

Maintenance updates

See Databricks Runtime 10.2 maintenance updates.

System environment

• Operating System: Ubuntu 20.04.3 LTS

• Java: Zulu 8.58.0.13-CA-linux64

• Scala: 2.12.14

• Python: 3.8.10

• R: 4.1.2

• Delta Lake: 1.1.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
Antergos Linux	2015.10 (ISO-Rolling)	appdirs	1.4.4	argon2-cffi	20.1.0
async-generator	1.10	attrs	20.3.0	backcall	0.2.0
bidict	0.21.4	bleach	3.3.0	boto3	1.16.7
botocore	1.19.7	certifi	2020.12.5	cffi	1.14.5
chardet	4.0.0	cycler	0.10.0	Cython	0.29.23
dbus-python	1.2.16	decorator	5.0.6	defusedxml	0.7.1
distlib	0.3.3	distro-info	0.23ubuntu1	entrypoints	0.3
facets-overview	1.0.0	filelock	3.4.0	idna	2.10
ipykernel	5.3.4	ipython	7.22.0	ipython-genutils	0.2.0
ipywidgets	7.6.3	jedi	0.17.2	Jinja2	2.11.3
jmespath	0.10.0	joblib	1.0.1	jsonschema	3.2.0
jupyter-client	6.1.12	jupyter-core	4.7.1	jupyterlab-pygments	0.1.2
jupyterlab-widgets	1.0.0	kiwisolver	1.3.1	koalas	1.8.2
MarkupSafe	2.0.1	matplotlib	3.4.2	mistune	0.8.4
nbclient	0.5.3	nbconvert	6.0.7	nbformat	5.1.3
nest-asyncio	1.5.1	notebook	6.3.0	numpy	1.19.2
packaging	20.9	pandas	1.2.4	pandocfilters	1.4.3
parso	0.7.0	patsy	0.5.1	pexpect	4.8.0
pickleshare	0.7.5	Pillow	8.2.0	pip	21.0.1
plotly	5.3.1	prometheus-client	0.10.1	prompt-toolkit	3.0.17
protobuf	3.17.2	psycopg2	2.8.5	ptyprocess	0.7.0
pyarrow	4.0.0	pycparser	2.20	Pygments	2.8.1
PyGObject	3.36.0	pyparsing	2.4.7	pyrsistent	0.17.3
python-apt	2.0.0+ubuntu0.20.4.6	python-dateutil	2.8.1	python-engineio	4.3.0
python-socketio	5.4.1	pytz	2020.5	pyzmq	20.0.0
requests	2.25.1	requests-unixsocket	0.2.0	s3transfer	0.3.7
scikit-learn	0.24.1	scipy	1.6.2	seaborn	0.11.1
Send2Trash	1.5.0	setuptools	52.0.0	six	1.15.0
ssh-import-id	5.10	statsmodels	0.12.2	tenacity	8.0.1
terminado	0.9.4	testpath	0.4.4	threadpoolctl	2.1.0
tornado	6.1	traitlets	5.0.5	unattended-upgrades	0.1
urllib3	1.25.11	virtualenv	20.4.1	wcwidth	0.2.5
webencodings	0.5.1	wheel	0.36.2	widgetsnbextension	3.5.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2021-11-30.

Library	Version	Library	Version	Library	Version
askpass	1.1	assertthat	0.2.1	backports	1.4.0
base	4.1.2	base64enc	0.1-3	bit	4.0.4
bit64	4.0.5	blob	1.2.2	boot	1.3-28
brew	1.0-6	brio	1.1.2	broom	0.7.10
bslib	0.3.1	cachem	1.0.6	callr	3.7.0
caret	6.0-90	cellranger	1.1.0	chron	2.3-56
class	7.3-19	cli	3.1.0	clipr	0.7.1
cluster	2.1.2	codetools	0.2-18	colorspace	2.0-2
commonmark	1.7	compiler	4.1.2	config	0.3.1
cpp11	0.4.1	crayon	1.4.2	credentials	1.3.2
curl	4.3.2	data.table	1.14.2	datasets	4.1.2
DBI	1.1.1	dbplyr	2.1.1	desc	1.4.0
devtools	2.4.2	diffobj	0.3.5	digest	0.6.28
dplyr	1.0.7	dtplyr	1.1.0	e1071	1.7-9
ellipsis	0.3.2	evaluate	0.14	fansi	0.5.0
farver	2.1.0	fastmap	1.1.0	fontawesome	0.2.2
forcats	0.5.1	foreach	1.5.1	foreign	0.8-81
forge	0.2.0	fs	1.5.0	future	1.23.0
future.apply	1.8.1	gargle	1.2.0	generics	0.1.1
gert	1.4.3	ggplot2	3.3.5	gh	1.3.0
gitcreds	0.1.1	glmnet	4.1-3	globals	0.14.0
glue	1.5.0	googledrive	2.0.0	googlesheets4	1.0.0
gower	0.2.2	graphics	4.1.2	grDevices	4.1.2
grid	4.1.2	gridExtra	2.3	gsubfn	0.7
gtable	0.3.0	haven	2.4.3	highr	0.9
hms	1.1.1	htmltools	0.5.2	htmlwidgets	1.5.4
httpuv	1.6.3	httr	1.4.2	hwriter	1.3.2
hwriterPlus	1.0-3	ids	1.0.1	ini	0.3.1
ipred	0.9-12	isoband	0.2.5	iterators	1.0.13
jquerylib	0.1.4	jsonlite	1.7.2	KernSmooth	2.23-20
knitr	1.36	labeling	0.4.2	later	1.3.0
lattice	0.20-45	lava	1.6.10	lifecycle	1.0.1
listenv	0.8.0	lubridate	1.8.0	magrittr	2.0.1
markdown	1.1	MASS	7.3-54	Matrix	1.3-4
memoise	2.0.1	methods	4.1.2	mgcv	1.8-38
mime	0.12	ModelMetrics	1.2.2.2	modelr	0.1.8
munsell	0.5.0	nlme	3.1-152	nnet	7.3-16
numDeriv	2016.8-1.1	openssl	1.4.5	parallel	4.1.2
parallelly	1.29.0	pillar	1.6.4	pkgbuild	1.2.0
pkgconfig	2.0.3	pkgload	1.2.3	plogr	0.2.0
plyr	1.8.6	praise	1.0.0	prettyunits	1.1.1
pROC	1.18.0	processx	3.5.2	prodlim	2019.11.13
progress	1.2.2	progressr	0.9.0	promises	1.2.0.1
proto	1.0.0	proxy	0.4-26	ps	1.6.0
purrr	0.3.4	r2d3	0.2.5	R6	2.5.1
randomForest	4.6-14	rappdirs	0.3.3	rcmdcheck	1.4.0
RColorBrewer	1.1-2	Rcpp	1.0.7	RcppEigen	0.3.3.9.1
readr	2.1.0	readxl	1.3.1	recipes	0.1.17
rematch	1.0.1	rematch2	2.1.2	remotes	2.4.1
reprex	2.0.1	reshape2	1.4.4	rlang	0.4.12
rmarkdown	2.11	RODBC	1.3-19	roxygen2	7.1.2
rpart	4.1-15	rprojroot	2.0.2	Rserve	1.8-10
RSQLite	2.2.8	rstudioapi	0.13	rversions	2.1.1
rvest	1.0.2	sass	0.4.0	scales	1.1.1
selectr	0.4-2	sessioninfo	1.2.1	shape	1.4.6
shiny	1.7.1	sourcetools	0.1.7	sparklyr	1.7.2
SparkR	3.2.0	spatial	7.3-11	splines	4.1.2
sqldf	0.4-11	SQUAREM	2021.1	stats	4.1.2
stats4	4.1.2	stringi	1.7.6	stringr	1.4.0
survival	3.2-13	sys	3.4	tcltk	4.1.2
TeachingDemos	2.10	testthat	3.1.0	tibble	3.1.6
tidyr	1.1.4	tidyselect	1.1.1	tidyverse	1.3.1
timeDate	3043.102	tinytex	0.35	tools	4.1.2
tzdb	0.2.0	usethis	2.1.3	utf8	1.2.2
utils	4.1.2	uuid	1.0-3	vctrs	0.3.8
viridisLite	0.4.0	vroom	1.5.6	waldo	0.3.1
whisker	0.4	withr	2.4.2	xfun	0.28
xml2	1.3.2	xopen	1.0.0	xtable	1.8-4
yaml	2.2.1	zip	2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.11.655
com.amazonaws	aws-java-sdk-cloudformation	1.11.655
com.amazonaws	aws-java-sdk-cloudfront	1.11.655
com.amazonaws	aws-java-sdk-cloudhsm	1.11.655
com.amazonaws	aws-java-sdk-cloudsearch	1.11.655
com.amazonaws	aws-java-sdk-cloudtrail	1.11.655
com.amazonaws	aws-java-sdk-cloudwatch	1.11.655
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.11.655
com.amazonaws	aws-java-sdk-codedeploy	1.11.655
com.amazonaws	aws-java-sdk-cognitoidentity	1.11.655
com.amazonaws	aws-java-sdk-cognitosync	1.11.655
com.amazonaws	aws-java-sdk-config	1.11.655
com.amazonaws	aws-java-sdk-core	1.11.655
com.amazonaws	aws-java-sdk-datapipeline	1.11.655
com.amazonaws	aws-java-sdk-directconnect	1.11.655
com.amazonaws	aws-java-sdk-directory	1.11.655
com.amazonaws	aws-java-sdk-dynamodb	1.11.655
com.amazonaws	aws-java-sdk-ec2	1.11.655
com.amazonaws	aws-java-sdk-ecs	1.11.655
com.amazonaws	aws-java-sdk-efs	1.11.655
com.amazonaws	aws-java-sdk-elasticache	1.11.655
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.11.655
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.11.655
com.amazonaws	aws-java-sdk-elastictranscoder	1.11.655
com.amazonaws	aws-java-sdk-emr	1.11.655
com.amazonaws	aws-java-sdk-glacier	1.11.655
com.amazonaws	aws-java-sdk-glue	1.11.655
com.amazonaws	aws-java-sdk-iam	1.11.655
com.amazonaws	aws-java-sdk-importexport	1.11.655
com.amazonaws	aws-java-sdk-kinesis	1.11.655
com.amazonaws	aws-java-sdk-kms	1.11.655
com.amazonaws	aws-java-sdk-lambda	1.11.655
com.amazonaws	aws-java-sdk-logs	1.11.655
com.amazonaws	aws-java-sdk-machinelearning	1.11.655
com.amazonaws	aws-java-sdk-opsworks	1.11.655
com.amazonaws	aws-java-sdk-rds	1.11.655
com.amazonaws	aws-java-sdk-redshift	1.11.655
com.amazonaws	aws-java-sdk-route53	1.11.655
com.amazonaws	aws-java-sdk-s3	1.11.655
com.amazonaws	aws-java-sdk-ses	1.11.655
com.amazonaws	aws-java-sdk-simpledb	1.11.655
com.amazonaws	aws-java-sdk-simpleworkflow	1.11.655
com.amazonaws	aws-java-sdk-sns	1.11.655
com.amazonaws	aws-java-sdk-sqs	1.11.655
com.amazonaws	aws-java-sdk-ssm	1.11.655
com.amazonaws	aws-java-sdk-storagegateway	1.11.655
com.amazonaws	aws-java-sdk-sts	1.11.655
com.amazonaws	aws-java-sdk-support	1.11.655
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.11.655
com.amazonaws	jmespath-java	1.11.655
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.12.3
com.fasterxml.jackson.core	jackson-core	2.12.3
com.fasterxml.jackson.core	jackson-databind	2.12.3
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.12.3
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.12.3
com.fasterxml.jackson.module	jackson-module-paranamer	2.12.3
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.12.3
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.luben	zstd-jni	1.5.0-4
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.8.6
com.google.crypto.tink	tink	1.6.0
com.google.flatbuffers	flatbuffers-java	1.9.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	1.4.195
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	9.2.1.jre8
com.ning	compress-lzf	1.0.3
com.sun.istack	istack-commons-runtime	3.0.8
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.2
commons-codec	commons-codec	1.15
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.8.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-net	commons-net	3.1
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	1.3.2
dev.ludovic.netlib	blas	1.3.2
dev.ludovic.netlib	lapack	1.3.2
hive-2.3__hadoop-3.2	jets3t-0.7	liball_deps_2.12
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.21
io.delta	delta-sharing-spark_2.12	0.2.0
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.68.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.annotation	javax.annotation-api	1.3.2
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.2
javax.xml.stream	stax-api	1.0-2
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.10
log4j	apache-log4j-extras	1.2.17
log4j	log4j	1.2.17
maven-trees	hive-2.3__hadoop-3.2	liball_deps_2.12
net.java.dev.jna	jna	5.8.0
net.razorvine	pyrolite	4.30
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.3
net.snowflake	spark-snowflake_2.12	2.9.0-spark_3.1
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	2.0.0
org.apache.arrow	arrow-memory-core	2.0.0
org.apache.arrow	arrow-memory-netty	2.0.0
org.apache.arrow	arrow-vector	2.0.0
org.apache.avro	avro	1.10.2
org.apache.avro	avro-ipc	1.10.2
org.apache.avro	avro-mapred	1.10.2
org.apache.commons	commons-compress	1.21
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.4.1
org.apache.commons	commons-text	1.6
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-api	3.3.1-databricks
org.apache.hadoop	hadoop-client-runtime	3.3.1
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.7.2
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.htrace	htrace-core4	4.1.0-incubating
org.apache.httpcomponents	httpclient	4.5.13
org.apache.httpcomponents	httpcore	4.4.12
org.apache.ivy	ivy	2.5.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.6.12
org.apache.orc	orc-mapreduce	1.6.12
org.apache.orc	orc-shims	1.6.12
org.apache.parquet	parquet-column	1.12.0-databricks-0003
org.apache.parquet	parquet-common	1.12.0-databricks-0003
org.apache.parquet	parquet-encoding	1.12.0-databricks-0003
org.apache.parquet	parquet-format-structures	1.12.0-databricks-0003
org.apache.parquet	parquet-hadoop	1.12.0-databricks-0003
org.apache.parquet	parquet-jackson	1.12.0-databricks-0003
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm9-shaded	4.20
org.apache.yetus	audience-annotations	0.5.0
org.apache.zookeeper	zookeeper	3.6.2
org.apache.zookeeper	zookeeper-jute	3.6.2
org.checkerframework	checker-qual	3.5.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.43.v20210629
org.eclipse.jetty	jetty-continuation	9.4.43.v20210629
org.eclipse.jetty	jetty-http	9.4.43.v20210629
org.eclipse.jetty	jetty-io	9.4.43.v20210629
org.eclipse.jetty	jetty-jndi	9.4.43.v20210629
org.eclipse.jetty	jetty-plus	9.4.43.v20210629
org.eclipse.jetty	jetty-proxy	9.4.43.v20210629
org.eclipse.jetty	jetty-security	9.4.43.v20210629
org.eclipse.jetty	jetty-server	9.4.43.v20210629
org.eclipse.jetty	jetty-servlet	9.4.43.v20210629
org.eclipse.jetty	jetty-servlets	9.4.43.v20210629
org.eclipse.jetty	jetty-util	9.4.43.v20210629
org.eclipse.jetty	jetty-util-ajax	9.4.43.v20210629
org.eclipse.jetty	jetty-webapp	9.4.43.v20210629
org.eclipse.jetty	jetty-xml	9.4.43.v20210629
org.eclipse.jetty.websocket	websocket-api	9.4.43.v20210629
org.eclipse.jetty.websocket	websocket-client	9.4.43.v20210629
org.eclipse.jetty.websocket	websocket-common	9.4.43.v20210629
org.eclipse.jetty.websocket	websocket-server	9.4.43.v20210629
org.eclipse.jetty.websocket	websocket-servlet	9.4.43.v20210629
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jaxb	jaxb-runtime	2.3.2
org.glassfish.jersey.containers	jersey-container-servlet	2.34
org.glassfish.jersey.containers	jersey-container-servlet-core	2.34
org.glassfish.jersey.core	jersey-client	2.34
org.glassfish.jersey.core	jersey-common	2.34
org.glassfish.jersey.core	jersey-server	2.34
org.glassfish.jersey.inject	jersey-hk2	2.34
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.7.1
org.mariadb.jdbc	mariadb-java-client	2.2.5
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.2.19
org.roaringbitmap	RoaringBitmap	0.9.14
org.roaringbitmap	shims	0.9.14
org.rocksdb	rocksdbjni	6.20.3
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.14
org.scala-lang	scala-library_2.12	2.12.14
org.scala-lang	scala-reflect_2.12	2.12.14
org.scala-lang.modules	scala-collection-compat_2.12	2.4.3
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.2
org.scalanlp	breeze_2.12	1.2
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.30
org.slf4j	jul-to-slf4j	1.7.30
org.slf4j	slf4j-api	1.7.30
org.slf4j	slf4j-log4j12	1.7.30
org.spark-project.spark	unused	1.0.0
org.springframework	spring-core	4.1.4.RELEASE
org.springframework	spring-test	4.1.4.RELEASE
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.8
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.4
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1