Databricks Runtime 11.3 LTS

The following release notes provide information about Databricks Runtime 11.3 LTS, powered by Apache Spark 3.3.0. Databricks released this version in October 2022.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

[Breaking change] New Python version requires updating Databricks Connect V1 Python clients

To apply required security patches, the Python version in Databricks Runtime 11.3 LTS is upgraded from 3.9.5 to 3.9.19. Because these changes might cause errors in clients that use specific PySpark functions, any clients that use Databricks Connect V1 for Python with Databricks Runtime 11.3 LTS must be updated to Python 3.9.7 or later.

New features and improvements

Structured Streaming trigger once is deprecated

The Trigger.Once setting has been deprecated. Databricks recommends you use Trigger.AvailableNow. See Configure Structured Streaming trigger intervals.

Change source path for Auto Loader

You can now change the directory input path for Auto Loader configured with directory listing mode without having to choose a new checkpoint directory. See Change source path for Auto Loader.

Databricks Kinesis connector now supports reading from Kinesis Data streams in EFO mode

You can now use the Databricks Kinesis structured streaming source in Databricks Runtime 11.3 LTS to run queries that read from Kinesis Data streams in enhanced fan-out mode. This allows dedicated throughput per shard, per consumer and record delivery in push mode.

New H3 geospatial functions and added Photon support for all H3 functions

Introducing 4 new H3 functions, h3_maxchild, h3_minchild, h3_pointash3, and h3_pointash3string. These functions are available in SQL, Scala, and Python. All H3 expressions are now supported in Photon. See H3 geospatial functions.

New features for Predictive I/O

Photon supports range mode for running frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. Photon also supports range mode for growing frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND offset_stop { PRECEDING | FOLLOWING }.

Increasing initial partitions to scan for selective queries

The value of initial partitions to scan has been increased to 10 for selective query with take/tail/limit in Photon-enabled clusters and LIMIT in Databricks SQL. With 10 partitions, you can avoid the overhead of launching multiple small jobs and a slow scale-up. You can also configure this through spark.sql.limit.selectiveInitialNumPartitions.

New AQE plan versions visualization

Introducing AQE plan versions that allows you to visualize your runtime plan updates from adaptive query execution (AQE).

New asynchronous progress tracking and log purging modes

Introducing Structured Streaming modes called asynchronous progress tracking and asynchronous log purging. Asynchronous log purging mode lowers the latency of streaming queries by removing logs used for progress tracking in the background.

Structured Streaming on Unity Catalog now supports display()

You can now use display() when you use Structured Streaming to work with tables registered in Unity Catalog.

Pipeline events are now logged in JSON format

Databricks now writes pipeline events to the driver log in JSON format. While each event will be JSON-parseable, large events may not contain all fields, or the fields may be truncated. Each event is logged in a single line with the prefix Event received: . The following is an example event.

Event received: {"id":"some-event-id","origin":{"pipeline_id":"some-pipeline-id","cluster_id":"some-cluster id"},"message":"simple [truncated] message","level":"WARN"}

Arbitrary Stateful Processing in Structured Streaming with Python

Introducing the applyInPandasWithState function that can be used to perform arbitrary stateful processing in PySpark. This is equivalent to the flatMapGroupsWithState function in the Java API.

Date inference in CSV files

Introducing improved inference of date type columns in CSV files. When the date format is consistent across the records for a column, those columns can be inferred as DateType. You can also have a combination of date formats across different columns. Databricks can automatically infer the date format for each column. Date columns in CSV files prior to Databricks Runtime 11.3 LTS are left as StringType.

Clone support for Apache Parquet and Apache Iceberg tables (Public Preview)

Clone can now be used to create and incrementally update Delta tables that mirror Apache Parquet and Apache Iceberg tables. You can update your source Parquet table and incrementally apply the changes to their cloned Delta table with the clone command. See Incrementally clone Parquet and Iceberg tables to Delta Lake.

Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables

You can now use the MANAGED LOCATION SQL command to specify a cloud storage location for managed tables at the catalog and schema levels. See CREATE CATALOG and CREATE SCHEMA.

Behavior changes

Databricks Connect 11.3.2

Databricks Connect client update 11.3.2 is now supported. See Databricks Connect and Databricks Connect release notes.

Updated Databricks Snowflake connector

The Databricks Snowflake connector has been updated to the latest version of code from the open-source repository, Snowflake Data Source for Apache Spark. It is now fully compatible with Databricks Runtime 11.3 LTS, including predicate pushdown and internal query plan pushdown while maintaining all of the features of the open-source version.

Hadoop cache for S3A is now disabled

The Hadoop cache (FileSystem Apache Hadoop Main 3.3.4 API) for S3A is now disabled. This is to align with other cloud storage connectors. For workloads that rely on file system caching, make sure that newly created file systems are supplied with the correct Hadoop configurations, including credential providers.

Delta Lake stats collection schema now matches column order in table schema definition

This change addresses a bug in the Delta Lake protocol where stats were not collected for columns due to a mismatch in DataFrame and table column ordering. In some cases, you might encounter write performance degradation due to stats collection on previously untracked fields. See Data skipping for Delta Lake.

applyInPandasWithState throws an error if the query has a shuffle after the operator

The operator applyInPandasWithState throws an error if the query has shuffle after the operator. This happens when either the user adds shuffle after the operation, or the optimizer or sink adds shuffle implicitly.

Library upgrades

• Upgraded Python libraries:

  • distlib from 0.3.5 to 0.3.6

• Upgraded R libraries:

  • broom from 1.0.0 to 1.0.1

  • callr from 3.7.1 to 3.7.2

  • dplyr from 1.0.9 to 1.0.10

  • dtplyr from 1.2.1 to 1.2.2

  • forcats from 0.5.1 to 0.5.2

  • future from 1.27.0 to 1.28.0

  • future.apply from 1.9.0 to 1.9.1

  • gert from 1.7.0 to 1.8.0

  • globals from 0.16.0 to 0.16.1

  • gtable from 0.3.0 to 0.3.1

  • haven from 2.5.0 to 2.5.1

  • hms from 1.1.1 to 1.1.2

  • httr from 1.4.3 to 1.4.4

  • knitr from 1.39 to 1.40

  • modelr from 0.1.8 to 0.1.9

  • pillar from 1.8.0 to 1.8.1

  • progressr from 0.10.1 to 0.11.0

  • readxl from 1.4.0 to 1.4.1

  • reprex from 2.0.1 to 2.0.2

  • rlang from 1.0.4 to 1.0.5

  • rmarkdown from 2.14 to 2.16

  • RSQLite from 2.2.15 to 2.2.16

  • rstudioapi from 0.13 to 0.14

  • rversions from 2.1.1 to 2.1.2

  • rvest from 1.0.2 to 1.0.3

  • scales from 1.2.0 to 1.2.1

  • sparklyr from 1.7.7 to 1.7.8

  • stringr from 1.4.0 to 1.4.1

  • survival from 3.2-13 to 3.4-0

  • tinytex from 0.40 to 0.41

  • viridisLite from 0.4.0 to 0.4.1

• Upgraded Java libraries:

  • com.fasterxml.jackson.core.jackson-annotations from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.core.jackson-core from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.core.jackson-databind from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.3 to 2.13.4

  • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.3 to 2.13.4

  • org.apache.hadoop.hadoop-client-api from 3.3.2-databricks to 3.3.4-databricks

  • org.apache.hadoop.hadoop-client-runtime from 3.3.2 to 3.3.4

  • org.apache.orc.orc-core from 1.7.5 to 1.7.6

  • org.apache.orc.orc-mapreduce from 1.7.5 to 1.7.6

  • org.apache.orc.orc-shims from 1.7.5 to 1.7.6

  • org.apache.parquet.parquet-column from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.apache.parquet.parquet-common from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.apache.parquet.parquet-encoding from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.apache.parquet.parquet-format-structures from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.apache.parquet.parquet-hadoop from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.apache.parquet.parquet-jackson from 1.12.0-databricks-0004 to 1.12.0-databricks-0007

  • org.glassfish.jersey.containers.jersey-container-servlet from 2.34 to 2.36

  • org.glassfish.jersey.containers.jersey-container-servlet-core from 2.34 to 2.36

  • org.glassfish.jersey.core.jersey-client from 2.34 to 2.36

  • org.glassfish.jersey.core.jersey-common from 2.34 to 2.36

  • org.glassfish.jersey.core.jersey-server from 2.34 to 2.36

  • org.glassfish.jersey.inject.jersey-hk2 from 2.34 to 2.36

Apache Spark

Databricks Runtime 11.3 LTS includes Apache Spark 3.3.0. This release includes all Spark fixes and improvements included in Databricks Runtime 11.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-39957] [WARMFIX][SC-111425][CORE] Delay onDisconnected to enable Driver receives ExecutorExitCode

• [SPARK-39955] [WARMFIX][SC-111424][CORE] Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

• [SPARK-40474] [SC-106248][Cherry-Pick] Correct CSV schema inference behavior for datetime columns and introduce auto detection for Date fields

• [SPARK-40535] [SC-111243][SQL] Fix bug the buffer of AggregatingAccumulator will not be created if the input rows is empty

• [SPARK-40434] [SC-111125][SC-111144][SC-111138][SPARK-40435][11.3][SS][PYTHON] Implement applyInPandasWithState in PySpark

• [SPARK-40460] [SC-110832][SS] Fix streaming metrics when selecting _metadata

• [SPARK-40324] [SC-109943][SQL] Provide a query context of ParseException

• [SPARK-40466] [SC-110899][SS] Improve the error message when DSv2 is disabled while DSv1 is not available

• [SPARK-40456] [SC-110848][SQL] PartitionIterator.hasNext should be cheap to call repeatedly

• [SPARK-40169] [SC-110772][SQL] Don’t pushdown Parquet filters with no reference to data schema

• [SPARK-40467] [SC-110759][SS] Split FlatMapGroupsWithState down to multiple test suites

• [SPARK-40468] [SC-110813][SQL] Fix column pruning in CSV when corruptrecord is selected

• [SPARK-40291] [SC-110085][SQL] Improve the message for column not in group by clause error

• [SPARK-40398] [SC-110762][CORE][SQL] Use Loop instead of Arrays.stream api

• [SPARK-40433] [SC-110684][SS][PYTHON] Add toJVMRow in PythonSQLUtils to convert pickled PySpark Row to JVM Row

• [SPARK-40414] [SC-110568][SQL][PYTHON] More generic type on PythonArrowInput and PythonArrowOutput

• [SPARK-40352] [SC-109945][SQL] Add function aliases: len, datepart, dateadd, date_diff and curdate

• [SPARK-40470] [SC-110761][SQL] Handle GetArrayStructFields and GetMapValue in “arrays_zip” function

• [SPARK-40387] [SC-110685][SQL] Improve the implementation of Spark Decimal

• [SPARK-40429] [SC-110675][SQL] Only set KeyGroupedPartitioning when the referenced column is in the output

• [SPARK-40432] [SC-110716][SS][PYTHON] Introduce GroupStateImpl and GroupStateTimeout in PySpark

• [SPARK-39915] [SC-110496][SQL] Ensure the output partitioning is user-specified in AQE

• [SPARK-29260] [SQL] Support ALTER DATABASE SET LOCATION if HMS supports

• [SPARK-40185] [SC-110056][SQL] Remove column suggestion when the candidate list is empty

• [SPARK-40362] [SC-110401][SQL] Fix BinaryComparison canonicalization

• [SPARK-40411] [SC-110381][SS] Refactor FlatMapGroupsWithStateExec to have a parent trait

• [SPARK-40293] [SC-110084][SQL] Make the V2 table error message more meaningful

• [SPARK-38734] [SC-110383][SQL] Remove the error class INDEX_OUT_OF_BOUNDS

• [SPARK-40292] [SC-110300][SQL] Fix column names in “arrays_zip” function when arrays are referenced from nested structs

• [SPARK-40276] [SC-109674][CORE] Reduce the result size of RDD.takeOrdered

• [SPARK-40197] [SC-109176][SQL] Replace query plan with context for MULTI_VALUE_SUBQUERY_ERROR

• [SPARK-40300] [SC-109942][SQL] Migrate onto the DATATYPE_MISMATCH error class

• [SPARK-40149] [SC-110055][SQL] Propagate metadata columns through Project

• [SPARK-40280] [SC-110146][SQL] Add support for parquet push down for annotated int and long

• [SPARK-40220] [SC-110143][SC-109175][SQL] Don’t output the empty map of error message parameters

• [SPARK-40295] [SC-110070][SQL] Allow v2 functions with literal args in write distribution/ordering

• [SPARK-40156] [SC-109264][SQL] url_decode() should the return an error class

• [SPARK-39195] [SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

• [SPARK-40260] [SC-109424][SQL] Use error classes in the compilation errors of GROUP BY a position

• [SPARK-40205] [SC-110144][SC-109082][SQL] Provide a query context of ELEMENT_AT_BY_INDEX_ZERO

• [SPARK-40112] [SC-109676][SQL] Improve the TO_BINARY() function

• [SPARK-40209] [SC-109081][SQL] Don’t change the interval value of Decimal in changePrecision() on errors

• [SPARK-40319] [SC-109873][SQL] Remove duplicated query execution error method for PARSE_DATETIME_BY_NEW_PARSER

• [SPARK-40222] [SC-109209][SQL] Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children

• [SPARK-40183] [SC-108907][SQL] Use error class NUMERIC_VALUE_OUT_OF_RANGE for overflow in decimal conversion

• [SPARK-40180] [SC-109069][SQL] Format error messages by spark-sql

• [SPARK-40153] [SC-109165][SQL] Unify resolve functions and table-valued functions

• [SPARK-40308] [SC-109880][SQL] Allow non-foldable delimiter arguments to str_to_map function

• [SPARK-40219] [SC-110052][SC-109663][SQL] Resolved view logical plan should hold the schema to avoid redundant lookup

• [SPARK-40098] [SC-109939][SC-108693][SQL] Format error messages in the Thrift Server

• [SPARK-39917] [SC-109038][SQL] Use different error classes for numeric/interval arithmetic overflow

• [SPARK-40033] [SC-109875][SQL] Nested schema pruning support through element_at

• [SPARK-40194] [SC-109660][SQL] SPLIT function on empty regex should truncate trailing empty string.

• [SPARK-40228] [SC-109835][SQL] Do not simplify multiLike if child is not a cheap expression

• [SPARK-40039] [SC-109896][SC-109260][SS] Introducing a streaming checkpoint file manager based on Hadoop’s Abortable interface

• [SPARK-40285] [SC-109679][SQL] Simplify the roundTo[Numeric] for Spark Decimal

• [SPARK-39896] [SC-109658][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed

• [SPARK-40040] [SC-109662][SQL] Push local limit to both sides if join condition is empty

• [SPARK-40055] [SC-109075][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog

• [SPARK-39915] [SC-109391][SQL] Dataset.repartition(N) may not create N partitions Non-AQE part

• [SPARK-40207] [SC-109401][SQL] Specify the column name when the data type is not supported by datasource

• [SPARK-40245] [SC-109295][SQL] Fix FileScan equality check when partition or data filter columns are not read

• [SPARK-40113] [SC-109405][SQL] Reactor ParquetScanBuilder DataSourceV2 interface implementations

• [SPARK-40211] [SC-109226][CORE][SQL] Allow customize initial partitions number in take() behavior

• [SPARK-40252] [SC-109379][SQL] Replace Stream.collect(Collectors.joining) with StringJoiner Api

• [SPARK-40247] [SC-109272][SQL] Fix BitSet equality check

• [SPARK-40067] [SQL] Use Table#name() instead of Scan#name() to populate the table name in the BatchScan node in SparkUI

• [SPARK-39966] [SQL] Use V2 Filter in SupportsDelete

• [SPARK-39607] [SC-109268][SQL][DSV2] Distribution and ordering support V2 function in writing

• [SPARK-40224] [SC-109271][SQL] Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based

• [SPARK-40013] [SQL] DS V2 expressions should have the default toString

• [SPARK-40214] [SC-109079][PYTHON][SQL] add ‘get’ to functions

• [SPARK-40192] [SC-109089][SQL][ML] Remove redundant groupby

• [SPARK-40146] [SC-108694][SQL] Simply the codegen of getting map value

• [SPARK-40109] [SQL] New SQL function: get()

• [SPARK-39929] [SQL] DS V2 supports push down string functions(non ANSI)

• [SPARK-39819] [SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions)

• [SPARK-40213] [SC-109077][SQL] Support ASCII value conversion for Latin-1 characters

• [SPARK-39887] [SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique

• [SPARK-39764] [SQL] Make PhysicalOperation the same as ScanOperation

• [SPARK-39964] [SQL] DS V2 pushdown should unify the translate path

• [SPARK-39528] [SQL] Use V2 Filter in SupportsRuntimeFiltering

• [SPARK-40066] [SQL] ANSI mode: always return null on invalid access to map column

• [SPARK-39912] [SPARK-39828][SQL] Refine CatalogImpl

• [SPARK-39833] [SC-108736][SQL] Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns

• [SPARK-39880] [SQL] V2 SHOW FUNCTIONS command should print qualified function name like v1

• [SPARK-39767] [SQL] Remove UnresolvedDBObjectName and add UnresolvedIdentifier

• [SPARK-40163] [SC-108740][SQL] feat: SparkSession.config(Map)

• [SPARK-40136] [SQL] Fix the fragment of SQL query contexts

• [SPARK-40107] [SC-108689][SQL] Pull out empty2null conversion from FileFormatWriter

• [SPARK-40121] [PYTHON][SQL] Initialize projection used for Python UDF

• [SPARK-40128] [SQL] Make the VectorizedColumnReader recognize DELTA_LENGTH_BYTE_ARRAY as a standalone column encoding

• [SPARK-40132] [ML] Restore rawPredictionCol to MultilayerPerceptronClassifier.setParams

• [SPARK-40050] [SC-108696][SQL] Enhance EliminateSorts to support removing sorts via LocalLimit

• [SPARK-39629] [SQL] Support v2 SHOW FUNCTIONS

• [SPARK-39925] [SC-108734][SQL] Add array_sort(column, comparator) overload to DataFrame operations

• [SPARK-40117] [PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite

• [SPARK-40105] [SQL] Improve repartition in ReplaceCTERefWithRepartition

• [SPARK-39503] [SQL] Add session catalog name for v1 database table and function

• [SPARK-39889] [SQL] Use different error classes for numeric/interval divided by 0

• [SPARK-39741] [SQL] Support url encode/decode as built-in function and tidy up url-related functions

• [SPARK-40102] [SQL] Use SparkException instead of IllegalStateException in SparkPlan

• [SPARK-40014] [SQL] Support cast of decimals to ANSI intervals

• [SPARK-39776] [SQL][FOLLOW] Update UT of PlanStabilitySuite in ANSI mode

• [SPARK-39963] [SQL] Simplify SimplifyCasts.isWiderCast

Maintenance updates

See Databricks Runtime 11.3 maintenance updates.

System environment

• Operating System: Ubuntu 20.04.5 LTS

• Java: Zulu 8.56.0.21-CA-linux64

• Scala: 2.12.14

• Python: 3.9.19

• R: 4.1.3

• Delta Lake: 2.1.0

Installed Python libraries

Library	Version	Library	Version	Library	Version
argon2-cffi	20.1.0	async-generator	1.10	attrs	21.2.0
backcall	0.2.0	backports.entry-points-selectable	1.1.1	black	22.3.0
bleach	4.0.0	boto3	1.21.18	botocore	1.24.18
certifi	2021.10.8	cffi	1.14.6	chardet	4.0.0
charset-normalizer	2.0.4	click	8.0.3	cryptography	3.4.8
cycler	0.10.0	Cython	0.29.24	dbus-python	1.2.16
debugpy	1.4.1	decorator	5.1.0	defusedxml	0.7.1
distlib	0.3.6	entrypoints	0.3	facets-overview	1.0.0
filelock	3.8.0	idna	3.2	ipykernel	6.12.1
ipython	7.32.0	ipython-genutils	0.2.0	ipywidgets	7.7.0
jedi	0.18.0	Jinja2	2.11.3	jmespath	0.10.0
joblib	1.0.1	jsonschema	3.2.0	jupyter-client	6.1.12
jupyter-core	4.8.1	jupyterlab-pygments	0.1.2	jupyterlab-widgets	1.0.0
kiwisolver	1.3.1	MarkupSafe	2.0.1	matplotlib	3.4.3
matplotlib-inline	0.1.2	mistune	0.8.4	mypy-extensions	0.4.3
nbclient	0.5.3	nbconvert	6.1.0	nbformat	5.1.3
nest-asyncio	1.5.1	notebook	6.4.5	numpy	1.20.3
packaging	21.0	pandas	1.3.4	pandocfilters	1.4.3
parso	0.8.2	pathspec	0.9.0	patsy	0.5.2
pexpect	4.8.0	pickleshare	0.7.5	Pillow	8.4.0
pip	21.2.4	platformdirs	2.5.2	plotly	5.9.0
prometheus-client	0.11.0	prompt-toolkit	3.0.20	protobuf	4.21.5
psutil	5.8.0	psycopg2	2.9.3	ptyprocess	0.7.0
pyarrow	7.0.0	pycparser	2.20	Pygments	2.10.0
PyGObject	3.36.0	pyodbc	4.0.31	pyparsing	3.0.4
pyrsistent	0.18.0	python-dateutil	2.8.2	pytz	2021.3
pyzmq	22.2.1	requests	2.26.0	requests-unixsocket	0.2.0
s3transfer	0.5.2	scikit-learn	0.24.2	scipy	1.7.1
seaborn	0.11.2	Send2Trash	1.8.0	setuptools	58.0.4
six	1.16.0	ssh-import-id	5.10	statsmodels	0.12.2
tenacity	8.0.1	terminado	0.9.4	testpath	0.5.0
threadpoolctl	2.2.0	tokenize-rt	4.2.1	tomli	2.0.1
tornado	6.1	traitlets	5.1.0	typing-extensions	3.10.0.2
unattended-upgrades	0.1	urllib3	1.26.7	virtualenv	20.8.0
wcwidth	0.2.5	webencodings	0.5.1	wheel	0.37.0
widgetsnbextension	3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-09-08.

Library	Version	Library	Version	Library	Version
askpass	1.1	assertthat	0.2.1	backports	1.4.1
base	4.1.3	base64enc	0.1-3	bit	4.0.4
bit64	4.0.5	blob	1.2.3	boot	1.3-28
brew	1.0-7	brio	1.1.3	broom	1.0.1
bslib	0.4.0	cachem	1.0.6	callr	3.7.2
caret	6.0-93	cellranger	1.1.0	chron	2.3-57
class	7.3-20	cli	3.3.0	clipr	0.8.0
cluster	2.1.3	codetools	0.2-18	colorspace	2.0-3
commonmark	1.8.0	compiler	4.1.3	config	0.3.1
cpp11	0.4.2	crayon	1.5.1	credentials	1.3.2
curl	4.3.2	data.table	1.14.2	datasets	4.1.3
DBI	1.1.3	dbplyr	2.2.1	desc	1.4.1
devtools	2.4.4	diffobj	0.3.5	digest	0.6.29
downlit	0.4.2	dplyr	1.0.10	dtplyr	1.2.2
e1071	1.7-11	ellipsis	0.3.2	evaluate	0.16
fansi	1.0.3	farver	2.1.1	fastmap	1.1.0
fontawesome	0.3.0	forcats	0.5.2	foreach	1.5.2
foreign	0.8-82	forge	0.2.0	fs	1.5.2
future	1.28.0	future.apply	1.9.1	gargle	1.2.0
generics	0.1.3	gert	1.8.0	ggplot2	3.3.6
gh	1.3.0	gitcreds	0.1.1	glmnet	4.1-4
globals	0.16.1	glue	1.6.2	googledrive	2.0.0
googlesheets4	1.0.1	gower	1.0.0	graphics	4.1.3
grDevices	4.1.3	grid	4.1.3	gridExtra	2.3
gsubfn	0.7	gtable	0.3.1	hardhat	1.2.0
haven	2.5.1	highr	0.9	hms	1.1.2
htmltools	0.5.3	htmlwidgets	1.5.4	httpuv	1.6.5
httr	1.4.4	ids	1.0.1	ini	0.3.1
ipred	0.9-13	isoband	0.2.5	iterators	1.0.14
jquerylib	0.1.4	jsonlite	1.8.0	KernSmooth	2.23-20
knitr	1.40	labeling	0.4.2	later	1.3.0
lattice	0.20-45	lava	1.6.10	lifecycle	1.0.1
listenv	0.8.0	lubridate	1.8.0	magrittr	2.0.3
markdown	1.1	MASS	7.3-56	Matrix	1.4-1
memoise	2.0.1	methods	4.1.3	mgcv	1.8-40
mime	0.12	miniUI	0.1.1.1	ModelMetrics	1.2.2.2
modelr	0.1.9	munsell	0.5.0	nlme	3.1-157
nnet	7.3-17	numDeriv	2016.8-1.1	openssl	2.0.2
parallel	4.1.3	parallelly	1.32.1	pillar	1.8.1
pkgbuild	1.3.1	pkgconfig	2.0.3	pkgdown	2.0.6
pkgload	1.3.0	plogr	0.2.0	plyr	1.8.7
praise	1.0.0	prettyunits	1.1.1	pROC	1.18.0
processx	3.7.0	prodlim	2019.11.13	profvis	0.3.7
progress	1.2.2	progressr	0.11.0	promises	1.2.0.1
proto	1.0.0	proxy	0.4-27	ps	1.7.1
purrr	0.3.4	r2d3	0.2.6	R6	2.5.1
ragg	1.2.2	randomForest	4.7-1.1	rappdirs	0.3.3
rcmdcheck	1.4.0	RColorBrewer	1.1-3	Rcpp	1.0.9
RcppEigen	0.3.3.9.2	readr	2.1.2	readxl	1.4.1
recipes	1.0.1	rematch	1.0.1	rematch2	2.1.2
remotes	2.4.2	reprex	2.0.2	reshape2	1.4.4
rlang	1.0.5	rmarkdown	2.16	RODBC	1.3-19
roxygen2	7.2.1	rpart	4.1.16	rprojroot	2.0.3
Rserve	1.8-11	RSQLite	2.2.16	rstudioapi	0.14
rversions	2.1.2	rvest	1.0.3	sass	0.4.2
scales	1.2.1	selectr	0.4-2	sessioninfo	1.2.2
shape	1.4.6	shiny	1.7.2	sourcetools	0.1.7
sparklyr	1.7.8	SparkR	3.3.0	spatial	7.3-11
splines	4.1.3	sqldf	0.4-11	SQUAREM	2021.1
stats	4.1.3	stats4	4.1.3	stringi	1.7.8
stringr	1.4.1	survival	3.4-0	sys	3.4
systemfonts	1.0.4	tcltk	4.1.3	testthat	3.1.4
textshaping	0.3.6	tibble	3.1.8	tidyr	1.2.0
tidyselect	1.1.2	tidyverse	1.3.2	timeDate	4021.104
tinytex	0.41	tools	4.1.3	tzdb	0.3.0
urlchecker	1.0.1	usethis	2.1.6	utf8	1.2.2
utils	4.1.3	uuid	1.1-0	vctrs	0.4.1
viridisLite	0.4.1	vroom	1.5.7	waldo	0.4.0
whisker	0.4	withr	2.5.0	xfun	0.32
xml2	1.3.3	xopen	1.0.0	xtable	1.8-4
yaml	2.3.5	zip	2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID	Artifact ID	Version
antlr	antlr	2.7.7
com.amazonaws	amazon-kinesis-client	1.12.0
com.amazonaws	aws-java-sdk-autoscaling	1.12.189
com.amazonaws	aws-java-sdk-cloudformation	1.12.189
com.amazonaws	aws-java-sdk-cloudfront	1.12.189
com.amazonaws	aws-java-sdk-cloudhsm	1.12.189
com.amazonaws	aws-java-sdk-cloudsearch	1.12.189
com.amazonaws	aws-java-sdk-cloudtrail	1.12.189
com.amazonaws	aws-java-sdk-cloudwatch	1.12.189
com.amazonaws	aws-java-sdk-cloudwatchmetrics	1.12.189
com.amazonaws	aws-java-sdk-codedeploy	1.12.189
com.amazonaws	aws-java-sdk-cognitoidentity	1.12.189
com.amazonaws	aws-java-sdk-cognitosync	1.12.189
com.amazonaws	aws-java-sdk-config	1.12.189
com.amazonaws	aws-java-sdk-core	1.12.189
com.amazonaws	aws-java-sdk-datapipeline	1.12.189
com.amazonaws	aws-java-sdk-directconnect	1.12.189
com.amazonaws	aws-java-sdk-directory	1.12.189
com.amazonaws	aws-java-sdk-dynamodb	1.12.189
com.amazonaws	aws-java-sdk-ec2	1.12.189
com.amazonaws	aws-java-sdk-ecs	1.12.189
com.amazonaws	aws-java-sdk-efs	1.12.189
com.amazonaws	aws-java-sdk-elasticache	1.12.189
com.amazonaws	aws-java-sdk-elasticbeanstalk	1.12.189
com.amazonaws	aws-java-sdk-elasticloadbalancing	1.12.189
com.amazonaws	aws-java-sdk-elastictranscoder	1.12.189
com.amazonaws	aws-java-sdk-emr	1.12.189
com.amazonaws	aws-java-sdk-glacier	1.12.189
com.amazonaws	aws-java-sdk-glue	1.12.189
com.amazonaws	aws-java-sdk-iam	1.12.189
com.amazonaws	aws-java-sdk-importexport	1.12.189
com.amazonaws	aws-java-sdk-kinesis	1.12.189
com.amazonaws	aws-java-sdk-kms	1.12.189
com.amazonaws	aws-java-sdk-lambda	1.12.189
com.amazonaws	aws-java-sdk-logs	1.12.189
com.amazonaws	aws-java-sdk-machinelearning	1.12.189
com.amazonaws	aws-java-sdk-opsworks	1.12.189
com.amazonaws	aws-java-sdk-rds	1.12.189
com.amazonaws	aws-java-sdk-redshift	1.12.189
com.amazonaws	aws-java-sdk-route53	1.12.189
com.amazonaws	aws-java-sdk-s3	1.12.189
com.amazonaws	aws-java-sdk-ses	1.12.189
com.amazonaws	aws-java-sdk-simpledb	1.12.189
com.amazonaws	aws-java-sdk-simpleworkflow	1.12.189
com.amazonaws	aws-java-sdk-sns	1.12.189
com.amazonaws	aws-java-sdk-sqs	1.12.189
com.amazonaws	aws-java-sdk-ssm	1.12.189
com.amazonaws	aws-java-sdk-storagegateway	1.12.189
com.amazonaws	aws-java-sdk-sts	1.12.189
com.amazonaws	aws-java-sdk-support	1.12.189
com.amazonaws	aws-java-sdk-swf-libraries	1.11.22
com.amazonaws	aws-java-sdk-workspaces	1.12.189
com.amazonaws	jmespath-java	1.12.189
com.chuusai	shapeless_2.12	2.3.3
com.clearspring.analytics	stream	2.9.6
com.databricks	Rserve	1.8-3
com.databricks	jets3t	0.7.1-0
com.databricks.scalapb	compilerplugin_2.12	0.4.15-10
com.databricks.scalapb	scalapb-runtime_2.12	0.4.15-10
com.esotericsoftware	kryo-shaded	4.0.2
com.esotericsoftware	minlog	1.3.0
com.fasterxml	classmate	1.3.4
com.fasterxml.jackson.core	jackson-annotations	2.13.4
com.fasterxml.jackson.core	jackson-core	2.13.4
com.fasterxml.jackson.core	jackson-databind	2.13.4
com.fasterxml.jackson.dataformat	jackson-dataformat-cbor	2.13.4
com.fasterxml.jackson.datatype	jackson-datatype-joda	2.13.4
com.fasterxml.jackson.datatype	jackson-datatype-jsr310	2.13.4
com.fasterxml.jackson.module	jackson-module-paranamer	2.13.4
com.fasterxml.jackson.module	jackson-module-scala_2.12	2.13.4
com.github.ben-manes.caffeine	caffeine	2.3.4
com.github.fommil	jniloader	1.1
com.github.fommil.netlib	core	1.1.2
com.github.fommil.netlib	native_ref-java	1.1
com.github.fommil.netlib	native_ref-java-natives	1.1
com.github.fommil.netlib	native_system-java	1.1
com.github.fommil.netlib	native_system-java-natives	1.1
com.github.fommil.netlib	netlib-native_ref-linux-x86_64-natives	1.1
com.github.fommil.netlib	netlib-native_system-linux-x86_64-natives	1.1
com.github.luben	zstd-jni	1.5.2-1
com.github.wendykierp	JTransforms	3.1
com.google.code.findbugs	jsr305	3.0.0
com.google.code.gson	gson	2.8.6
com.google.crypto.tink	tink	1.6.1
com.google.flatbuffers	flatbuffers-java	1.12.0
com.google.guava	guava	15.0
com.google.protobuf	protobuf-java	2.6.1
com.h2database	h2	2.0.204
com.helger	profiler	1.1.1
com.jcraft	jsch	0.1.50
com.jolbox	bonecp	0.8.0.RELEASE
com.lihaoyi	sourcecode_2.12	0.1.9
com.microsoft.azure	azure-data-lake-store-sdk	2.3.9
com.microsoft.sqlserver	mssql-jdbc	9.2.1.jre8
com.ning	compress-lzf	1.1
com.sun.mail	javax.mail	1.5.2
com.tdunning	json	1.8
com.thoughtworks.paranamer	paranamer	2.8
com.trueaccord.lenses	lenses_2.12	0.4.12
com.twitter	chill-java	0.10.0
com.twitter	chill_2.12	0.10.0
com.twitter	util-app_2.12	7.1.0
com.twitter	util-core_2.12	7.1.0
com.twitter	util-function_2.12	7.1.0
com.twitter	util-jvm_2.12	7.1.0
com.twitter	util-lint_2.12	7.1.0
com.twitter	util-registry_2.12	7.1.0
com.twitter	util-stats_2.12	7.1.0
com.typesafe	config	1.2.1
com.typesafe.scala-logging	scala-logging_2.12	3.7.2
com.uber	h3	3.7.0
com.univocity	univocity-parsers	2.9.1
com.zaxxer	HikariCP	4.0.3
commons-cli	commons-cli	1.5.0
commons-codec	commons-codec	1.15
commons-collections	commons-collections	3.2.2
commons-dbcp	commons-dbcp	1.4
commons-fileupload	commons-fileupload	1.3.3
commons-httpclient	commons-httpclient	3.1
commons-io	commons-io	2.11.0
commons-lang	commons-lang	2.6
commons-logging	commons-logging	1.1.3
commons-pool	commons-pool	1.5.4
dev.ludovic.netlib	arpack	2.2.1
dev.ludovic.netlib	blas	2.2.1
dev.ludovic.netlib	lapack	2.2.1
info.ganglia.gmetric4j	gmetric4j	1.0.10
io.airlift	aircompressor	0.21
io.delta	delta-sharing-spark_2.12	0.5.1
io.dropwizard.metrics	metrics-core	4.1.1
io.dropwizard.metrics	metrics-graphite	4.1.1
io.dropwizard.metrics	metrics-healthchecks	4.1.1
io.dropwizard.metrics	metrics-jetty9	4.1.1
io.dropwizard.metrics	metrics-jmx	4.1.1
io.dropwizard.metrics	metrics-json	4.1.1
io.dropwizard.metrics	metrics-jvm	4.1.1
io.dropwizard.metrics	metrics-servlets	4.1.1
io.netty	netty-all	4.1.74.Final
io.netty	netty-buffer	4.1.74.Final
io.netty	netty-codec	4.1.74.Final
io.netty	netty-common	4.1.74.Final
io.netty	netty-handler	4.1.74.Final
io.netty	netty-resolver	4.1.74.Final
io.netty	netty-tcnative-classes	2.0.48.Final
io.netty	netty-transport	4.1.74.Final
io.netty	netty-transport-classes-epoll	4.1.74.Final
io.netty	netty-transport-classes-kqueue	4.1.74.Final
io.netty	netty-transport-native-epoll-linux-aarch_64	4.1.74.Final
io.netty	netty-transport-native-epoll-linux-x86_64	4.1.74.Final
io.netty	netty-transport-native-kqueue-osx-aarch_64	4.1.74.Final
io.netty	netty-transport-native-kqueue-osx-x86_64	4.1.74.Final
io.netty	netty-transport-native-unix-common	4.1.74.Final
io.prometheus	simpleclient	0.7.0
io.prometheus	simpleclient_common	0.7.0
io.prometheus	simpleclient_dropwizard	0.7.0
io.prometheus	simpleclient_pushgateway	0.7.0
io.prometheus	simpleclient_servlet	0.7.0
io.prometheus.jmx	collector	0.12.0
jakarta.annotation	jakarta.annotation-api	1.3.5
jakarta.servlet	jakarta.servlet-api	4.0.3
jakarta.validation	jakarta.validation-api	2.0.2
jakarta.ws.rs	jakarta.ws.rs-api	2.1.6
javax.activation	activation	1.1.1
javax.annotation	javax.annotation-api	1.3.2
javax.el	javax.el-api	2.2.4
javax.jdo	jdo-api	3.0.1
javax.transaction	jta	1.1
javax.transaction	transaction-api	1.1
javax.xml.bind	jaxb-api	2.2.11
javolution	javolution	5.5.1
jline	jline	2.14.6
joda-time	joda-time	2.10.13
net.java.dev.jna	jna	5.8.0
net.razorvine	pickle	1.2
net.sf.jpam	jpam	1.1
net.sf.opencsv	opencsv	2.3
net.sf.supercsv	super-csv	2.2.0
net.snowflake	snowflake-ingest-sdk	0.9.6
net.snowflake	snowflake-jdbc	3.13.14
net.sourceforge.f2j	arpack_combined_all	0.1
org.acplt.remotetea	remotetea-oncrpc	1.1.2
org.antlr	ST4	4.0.4
org.antlr	antlr-runtime	3.5.2
org.antlr	antlr4-runtime	4.8
org.antlr	stringtemplate	3.2.1
org.apache.ant	ant	1.9.2
org.apache.ant	ant-jsch	1.9.2
org.apache.ant	ant-launcher	1.9.2
org.apache.arrow	arrow-format	7.0.0
org.apache.arrow	arrow-memory-core	7.0.0
org.apache.arrow	arrow-memory-netty	7.0.0
org.apache.arrow	arrow-vector	7.0.0
org.apache.avro	avro	1.11.0
org.apache.avro	avro-ipc	1.11.0
org.apache.avro	avro-mapred	1.11.0
org.apache.commons	commons-collections4	4.4
org.apache.commons	commons-compress	1.21
org.apache.commons	commons-crypto	1.1.0
org.apache.commons	commons-lang3	3.12.0
org.apache.commons	commons-math3	3.6.1
org.apache.commons	commons-text	1.9
org.apache.curator	curator-client	2.13.0
org.apache.curator	curator-framework	2.13.0
org.apache.curator	curator-recipes	2.13.0
org.apache.derby	derby	10.14.2.0
org.apache.hadoop	hadoop-client-api	3.3.4-databricks
org.apache.hadoop	hadoop-client-runtime	3.3.4
org.apache.hive	hive-beeline	2.3.9
org.apache.hive	hive-cli	2.3.9
org.apache.hive	hive-jdbc	2.3.9
org.apache.hive	hive-llap-client	2.3.9
org.apache.hive	hive-llap-common	2.3.9
org.apache.hive	hive-serde	2.3.9
org.apache.hive	hive-shims	2.3.9
org.apache.hive	hive-storage-api	2.7.2
org.apache.hive.shims	hive-shims-0.23	2.3.9
org.apache.hive.shims	hive-shims-common	2.3.9
org.apache.hive.shims	hive-shims-scheduler	2.3.9
org.apache.httpcomponents	httpclient	4.5.13
org.apache.httpcomponents	httpcore	4.4.14
org.apache.ivy	ivy	2.5.0
org.apache.logging.log4j	log4j-1.2-api	2.18.0
org.apache.logging.log4j	log4j-api	2.18.0
org.apache.logging.log4j	log4j-core	2.18.0
org.apache.logging.log4j	log4j-slf4j-impl	2.18.0
org.apache.mesos	mesos-shaded-protobuf	1.4.0
org.apache.orc	orc-core	1.7.6
org.apache.orc	orc-mapreduce	1.7.6
org.apache.orc	orc-shims	1.7.6
org.apache.parquet	parquet-column	1.12.0-databricks-0007
org.apache.parquet	parquet-common	1.12.0-databricks-0007
org.apache.parquet	parquet-encoding	1.12.0-databricks-0007
org.apache.parquet	parquet-format-structures	1.12.0-databricks-0007
org.apache.parquet	parquet-hadoop	1.12.0-databricks-0007
org.apache.parquet	parquet-jackson	1.12.0-databricks-0007
org.apache.thrift	libfb303	0.9.3
org.apache.thrift	libthrift	0.12.0
org.apache.xbean	xbean-asm9-shaded	4.20
org.apache.yetus	audience-annotations	0.5.0
org.apache.zookeeper	zookeeper	3.6.2
org.apache.zookeeper	zookeeper-jute	3.6.2
org.checkerframework	checker-qual	3.5.0
org.codehaus.jackson	jackson-core-asl	1.9.13
org.codehaus.jackson	jackson-mapper-asl	1.9.13
org.codehaus.janino	commons-compiler	3.0.16
org.codehaus.janino	janino	3.0.16
org.datanucleus	datanucleus-api-jdo	4.2.4
org.datanucleus	datanucleus-core	4.1.17
org.datanucleus	datanucleus-rdbms	4.1.19
org.datanucleus	javax.jdo	3.2.0-m3
org.eclipse.jetty	jetty-client	9.4.46.v20220331
org.eclipse.jetty	jetty-continuation	9.4.46.v20220331
org.eclipse.jetty	jetty-http	9.4.46.v20220331
org.eclipse.jetty	jetty-io	9.4.46.v20220331
org.eclipse.jetty	jetty-jndi	9.4.46.v20220331
org.eclipse.jetty	jetty-plus	9.4.46.v20220331
org.eclipse.jetty	jetty-proxy	9.4.46.v20220331
org.eclipse.jetty	jetty-security	9.4.46.v20220331
org.eclipse.jetty	jetty-server	9.4.46.v20220331
org.eclipse.jetty	jetty-servlet	9.4.46.v20220331
org.eclipse.jetty	jetty-servlets	9.4.46.v20220331
org.eclipse.jetty	jetty-util	9.4.46.v20220331
org.eclipse.jetty	jetty-util-ajax	9.4.46.v20220331
org.eclipse.jetty	jetty-webapp	9.4.46.v20220331
org.eclipse.jetty	jetty-xml	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-api	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-client	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-common	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-server	9.4.46.v20220331
org.eclipse.jetty.websocket	websocket-servlet	9.4.46.v20220331
org.fusesource.leveldbjni	leveldbjni-all	1.8
org.glassfish.hk2	hk2-api	2.6.1
org.glassfish.hk2	hk2-locator	2.6.1
org.glassfish.hk2	hk2-utils	2.6.1
org.glassfish.hk2	osgi-resource-locator	1.0.3
org.glassfish.hk2.external	aopalliance-repackaged	2.6.1
org.glassfish.hk2.external	jakarta.inject	2.6.1
org.glassfish.jersey.containers	jersey-container-servlet	2.36
org.glassfish.jersey.containers	jersey-container-servlet-core	2.36
org.glassfish.jersey.core	jersey-client	2.36
org.glassfish.jersey.core	jersey-common	2.36
org.glassfish.jersey.core	jersey-server	2.36
org.glassfish.jersey.inject	jersey-hk2	2.36
org.hibernate.validator	hibernate-validator	6.1.0.Final
org.javassist	javassist	3.25.0-GA
org.jboss.logging	jboss-logging	3.3.2.Final
org.jdbi	jdbi	2.63.1
org.jetbrains	annotations	17.0.0
org.joda	joda-convert	1.7
org.jodd	jodd-core	3.5.2
org.json4s	json4s-ast_2.12	3.7.0-M11
org.json4s	json4s-core_2.12	3.7.0-M11
org.json4s	json4s-jackson_2.12	3.7.0-M11
org.json4s	json4s-scalap_2.12	3.7.0-M11
org.lz4	lz4-java	1.8.0
org.mariadb.jdbc	mariadb-java-client	2.7.4
org.mlflow	mlflow-spark	1.27.0
org.objenesis	objenesis	2.5.1
org.postgresql	postgresql	42.3.3
org.roaringbitmap	RoaringBitmap	0.9.25
org.roaringbitmap	shims	0.9.25
org.rocksdb	rocksdbjni	6.24.2
org.rosuda.REngine	REngine	2.1.0
org.scala-lang	scala-compiler_2.12	2.12.14
org.scala-lang	scala-library_2.12	2.12.14
org.scala-lang	scala-reflect_2.12	2.12.14
org.scala-lang.modules	scala-collection-compat_2.12	2.4.3
org.scala-lang.modules	scala-parser-combinators_2.12	1.1.2
org.scala-lang.modules	scala-xml_2.12	1.2.0
org.scala-sbt	test-interface	1.0
org.scalacheck	scalacheck_2.12	1.14.2
org.scalactic	scalactic_2.12	3.0.8
org.scalanlp	breeze-macros_2.12	1.2
org.scalanlp	breeze_2.12	1.2
org.scalatest	scalatest_2.12	3.0.8
org.slf4j	jcl-over-slf4j	1.7.36
org.slf4j	jul-to-slf4j	1.7.36
org.slf4j	slf4j-api	1.7.36
org.spark-project.spark	unused	1.0.0
org.threeten	threeten-extra	1.5.0
org.tukaani	xz	1.8
org.typelevel	algebra_2.12	2.0.1
org.typelevel	cats-kernel_2.12	2.1.1
org.typelevel	macro-compat_2.12	1.1.1
org.typelevel	spire-macros_2.12	0.17.0
org.typelevel	spire-platform_2.12	0.17.0
org.typelevel	spire-util_2.12	0.17.0
org.typelevel	spire_2.12	0.17.0
org.wildfly.openssl	wildfly-openssl	1.0.7.Final
org.xerial	sqlite-jdbc	3.8.11.2
org.xerial.snappy	snappy-java	1.1.8.4
org.yaml	snakeyaml	1.24
oro	oro	2.0.8
pl.edu.icm	JLargeArrays	1.5
software.amazon.ion	ion-java	1.0.2
stax	stax-api	1.0.1