Databricks Runtime 12.0 (EoS)

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

The following release notes provide information about Databricks Runtime 12.0, powered by Apache Spark 3.3.1.

Databricks released this version in December 2022.

New features and improvements

Predictive I/O is GA

Predictive I/O is now generally available. For more information, see What is predictive I/O?.

Photon writer supports zstd compression

Photon native writer now supports the zstd compression codec when zstd is enabled by setting spark.sql.parquet.compression.codec to zstd.

Support for stage-level task resource profile for standalone clusters

You can now use stage-level scheduling on standalone clusters when dynamic allocation is disabled. To use this feature, specify the task resources with ResourceProfileBuilder for each stage.

SQL support for selective overwrite with REPLACE WHERE

You can now selectively overwrite data matching an arbitrary expression in a Delta table using the following pattern.

INSERT INTO table_name REPLACE WHERE predicate append_relation

See Arbitrary selective overwrite with replaceWhere.

Watermarks are now supported in SQL

You can now specify watermarks using the Delta Live Tables SQL interface and in SQL queries against streaming DataFrames. See WATERMARK clause.

PySpark memory profiling

Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code in a user-defined function.

Dynamic pruning for DELETE and UPDATE

When using Photon-enabled compute, DELETE and UPDATE now use dynamic file and partition pruning where it improves performance. For example, dynamic pruning is enabled when a smaller source table is used to update or delete rows in a larger table.

Row-level delete metrics for partitioned data manipulation language operations

With partitioned predicates, users can now audit how many rows are deleted when running data manipulation language (DML) operations such as DELETE, TRUNCATE, and replaceWhere.

Bug fixes

Fixed an issue with JSON parsing in Auto Loader where all columns were left as strings. Previously, cloudFiles.inferColumnTypes was not set or set to false and the JSON contained nested objects.

Library upgrades

  • Upgraded Python libraries:

    • argon2-cffi from 20.1.0 to 21.3.0

    • attrs from 21.2.0 to 21.4.0

    • backports.entry-points-selectable from 1.1.1 to 1.2.0

    • bleach from 4.0.0 to 4.1.0

    • boto3 from 1.21.18 to 1.21.32

    • botocore from 1.24.18 to 1.24.32

    • cffi from 1.14.6 to 1.15.0

    • click from 8.0.3 to 8.0.4

    • cycler from 0.10.0 to 0.11.0

    • Cython from 0.29.24 to 0.29.28

    • debugpy from 1.4.1 to 1.5.1

    • decorator from 5.1.0 to 5.1.1

    • entrypoints from 0.3 to 0.4

    • idna from 3.2 to 3.3

    • ipykernel from 6.12.1 to 6.15.3

    • ipython from 7.32.0 to 8.5.0

    • ipywidgets from 7.7.0 to 7.7.2

    • jedi from 0.18.0 to 0.18.1

    • joblib from 1.0.1 to 1.1.0

    • jsonschema from 3.2.0 to 4.4.0

    • kiwisolver from 1.3.1 to 1.3.2

    • matplotlib from 3.4.3 to 3.5.1

    • nbclient from 0.5.3 to 0.5.13

    • nbconvert from 6.1.0 to 6.4.4

    • nbformat from 5.1.3 to 5.3.0

    • nest-asyncio from 1.5.1 to 1.5.5

    • notebook from 6.4.5 to 6.4.8

    • numpy from 1.20.3 to 1.21.5

    • packaging from 21.0 to 21.3

    • pandas from 1.3.4 to 1.4.2

    • pandocfilters from 1.4.3 to 1.5.0

    • parso from 0.8.2 to 0.8.3

    • Pillow from 8.4.0 to 9.0.1

    • platformdirs from 2.5.2 to 2.5.4

    • plotly from 5.9.0 to 5.6.0

    • prometheus-client from 0.11.0 to 0.13.1

    • protobuf from 4.21.5 to 3.19.4

    • pycparser from 2.20 to 2.21

    • Pygments from 2.10.0 to 2.11.2

    • pyodbc from 4.0.31 to 4.0.32

    • pyzmq from 22.2.1 to 22.3.0

    • requests from 2.26.0 to 2.27.1

    • s3transfer from 0.5.2 to 0.5.0

    • scikit-learn from 0.24.2 to 1.0.2

    • scipy from 1.7.1 to 1.7.3

    • statsmodels from 0.12.2 to 0.13.2

    • terminado from 0.9.4 to 0.13.1

    • tomli from 2.0.1 to 1.2.2

    • traitlets from 5.1.0 to 5.1.1

    • urllib3 from 1.26.7 to 1.26.9

    • widgetsnbextension from 3.6.0 to 3.6.1

  • Upgraded R libraries:

    • base from 4.1.3 to 4.2.2

    • brew from 1.0-7 to 1.0-8

    • bslib from 0.4.0 to 0.4.1

    • callr from 3.7.2 to 3.7.3

    • chron from 2.3-57 to 2.3-58

    • cli from 3.3.0 to 3.4.1

    • cluster from 2.1.3 to 2.1.4

    • commonmark from 1.8.0 to 1.8.1

    • compiler from 4.1.3 to 4.2.2

    • cpp11 from 0.4.2 to 0.4.3

    • crayon from 1.5.1 to 1.5.2

    • curl from 4.3.2 to 4.3.3

    • data.table from 1.14.2 to 1.14.4

    • datasets from 4.1.3 to 4.2.2

    • desc from 1.4.1 to 1.4.2

    • devtools from 2.4.4 to 2.4.5

    • digest from 0.6.29 to 0.6.30

    • e1071 from 1.7-11 to 1.7-12

    • evaluate from 0.16 to 0.18

    • fontawesome from 0.3.0 to 0.4.0

    • future from 1.28.0 to 1.29.0

    • future.apply from 1.9.1 to 1.10.0

    • gargle from 1.2.0 to 1.2.1

    • gert from 1.8.0 to 1.9.1

    • ggplot2 from 3.3.6 to 3.4.0

    • gh from 1.3.0 to 1.3.1

    • gitcreds from 0.1.1 to 0.1.2

    • graphics from 4.1.3 to 4.2.2

    • grDevices from 4.1.3 to 4.2.2

    • grid from 4.1.3 to 4.2.2

    • httpuv from 1.6.5 to 1.6.6

    • isoband from 0.2.5 to 0.2.6

    • jsonlite from 1.8.0 to 1.8.3

    • lava from 1.6.10 to 1.7.0

    • lifecycle from 1.0.1 to 1.0.3

    • lubridate from 1.8.0 to 1.9.0

    • markdown from 1.1 to 1.3

    • MASS from 7.3-56 to 7.3-58

    • Matrix from 1.4-1 to 1.5-1

    • methods from 4.1.3 to 4.2.2

    • mgcv from 1.8-40 to 1.8-41

    • nlme from 3.1-157 to 3.1-160

    • nnet from 7.3-17 to 7.3-18

    • openssl from 2.0.2 to 2.0.4

    • parallel from 4.1.3 to 4.2.2

    • pkgload from 1.3.0 to 1.3.1

    • processx from 3.7.0 to 3.8.0

    • ps from 1.7.1 to 1.7.2

    • purrr from 0.3.4 to 0.3.5

    • ragg from 1.2.2 to 1.2.4

    • RcppEigen from 0.3.3.9.2 to 0.3.3.9.3

    • readr from 2.1.2 to 2.1.3

    • recipes from 1.0.1 to 1.0.3

    • rlang from 1.0.5 to 1.0.6

    • rmarkdown from 2.16 to 2.18

    • rpart from 4.1.16 to 4.1.19

    • RSQLite from 2.2.16 to 2.2.18

    • shiny from 1.7.2 to 1.7.3

    • SparkR from 3.3.0 to 3.3.1

    • splines from 4.1.3 to 4.2.2

    • stats from 4.1.3 to 4.2.2

    • stats4 from 4.1.3 to 4.2.2

    • sys from 3.4 to 3.4.1

    • tcltk from 4.1.3 to 4.2.2

    • testthat from 3.1.4 to 3.1.5

    • tidyr from 1.2.0 to 1.2.1

    • tidyselect from 1.1.2 to 1.2.0

    • timeDate from 4021.104 to 4021.106

    • tinytex from 0.41 to 0.42

    • tools from 4.1.3 to 4.2.2

    • utils from 4.1.3 to 4.2.2

    • vctrs from 0.4.1 to 0.5.0

    • vroom from 1.5.7 to 1.6.0

    • xfun from 0.32 to 0.34

    • yaml from 2.3.5 to 2.3.6

    • zip from 2.2.0 to 2.2.2

  • Upgraded Java libraries:

    • com.fasterxml.jackson.core.jackson-databind from 2.13.4 to 2.13.4.2

    • org.apache.parquet.parquet-column from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.parquet.parquet-common from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.parquet.parquet-encoding from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.parquet.parquet-format-structures from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.parquet.parquet-hadoop from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.parquet.parquet-jackson from 1.12.0-databricks-0007 to 1.12.3-databricks-0001

    • org.apache.yetus.audience-annotations from 0.5.0 to 0.13.0

Apache Spark

Databricks Runtime 12.0 includes Apache Spark 3.3.1. This release includes all Spark fixes and improvements included in Databricks Runtime 11.3 LTS, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-40844] [SC-116996][12.x][12.0][12.0.0] Revert “[SC-113542][SS] Flip the default value of Kafka offset fetching config”

  • [SPARK-40646] [SC-116061] Revert “[SC-113379][SQL] Fix returning partial results in JSON data source and JSON functions”

  • [SPARK-41195] [SC-116305][SQL] Support PIVOT/UNPIVOT with join children

  • [SPARK-41178] [SC-116139][SQL] Fix parser rule precedence between JOIN and comma

  • [SPARK-41072] [SC-116140][SC-116044][SC-115852][SQL][SS] Add the error class STREAM_FAILED to StreamingQueryException

  • [SPARK-40921] Revert “[SC-115140][SC-114585][SQL] Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO

  • [SPARK-37980] [SC-115758] [SQL] Extend METADATA column to support row indexes for Parquet files

  • [SPARK-41055] [SC-115582][SQL] Rename _LEGACY_ERROR_TEMP_2424 to GROUP_BY_AGGREGATE

  • [SPARK-41101] [SC-115849][PYTHON][PROTOBUF] Message classname support for PYSPARK-PROTOBUF

  • [SPARK-40956] [SC-115867] SQL Equivalent for Dataframe overwrite command

  • [SPARK-41095] [SC-115659][SQL] Convert unresolved operators to internal errors

  • [SPARK-41144] [SC-115866][SQL] Unresolved hint should not cause query failure

  • [SPARK-41137] [SC-115858][SQL] Rename LATERAL_JOIN_OF_TYPE to INVALID_LATERAL_JOIN_TYPE

  • [SPARK-40998] [SC-114754][SQL] Rename the error class _LEGACY_ERROR_TEMP_0040 to INVALID_IDENTIFIER

  • [SPARK-41015] [SC-115130][SQL][PROTOBUF] UnitTest null check for data generator

  • [SPARK-40769] [SC-114923][CORE][SQL] Migrate type check failures of aggregate expressions onto error classes

  • [SPARK-41043] [SC-115183][SQL] Rename the error class _LEGACY_ERROR_TEMP_2429 to NUM_COLUMNS_MISMATCH

  • [SPARK-40777] [SC-114855][SQL][PROTOBUF] Protobuf import support and move error-classes.

  • [SPARK-41019] [SC-114987][SQL] Provide a query context to failAnalysis()

  • [SPARK-41027] [SC-115131][SQL] Use UNEXPECTED_INPUT_TYPE instead of MAP_FROM_ENTRIES_WRONG_TYPE

  • [SPARK-40476] [SC-115868][SC-111163][ML][SQL] Reduce the shuffle size of ALS

  • [SPARK-41134] [SC-115851][SQL] Improve error message of internal errors

  • [SPARK-37980] Revert ” [SQL] Extend METADATA column to support row indexes for Parquet files

  • [SPARK-41109] [SC-115788][SQL] Rename the error class LEGACYERROR_TEMP_1216 to INVALID_LIKE_PATTERN

  • [SPARK-40984] [SC-115844][CORE][SQL] Use NON_FOLDABLE_INPUT instead of FRAME_LESS_OFFSET_WITHOUT_FOLDABLE

  • [SPARK-41029] [SC-115828][SQL] Optimize constructor use of GenericArrayData for Scala 2.13

  • [SPARK-40697] [SC-113648][SQL] Add read-side char padding to cover external data files

  • [SPARK-40978] [SC-114574][SQL] Migrate failAnalysis() w/o a context onto error classes

  • [SPARK-40663] [SC-114586][SQL] Migrate execution errors onto error classes: LEGACYERROR_TEMP_2251-2275

  • [SPARK-38959] [SC-115672][SC-112707][SQL] DS V2: Support runtime group filtering in row-level commands

  • [SPARK-40372] [SC-115668][SQL] Migrate failures of array type checks onto error classes

  • [SPARK-41038] [SC-115371][SQL] Rename MULTI_VALUE_SUBQUERY_ERROR to SCALAR_SUBQUERY_TOO_MANY_ROWS

  • [SPARK-40965] [SC-114443][SQL] Rename the error class _LEGACY_ERROR_TEMP_1208 to FIELD_NOT_FOUND

  • [SPARK-40748] [SC-114696][SQL] Migrate type check failures of conditions onto error classes

  • [SPARK-40371] [SC-114557][SQL] Migrate type check failures of NthValue and NTile onto error classes

  • [SPARK-41092] [SC-115547][SQL] Do not use identifier to match interval units

  • [SPARK-41009] [SC-115360][SQL] Rename the error class _LEGACY_ERROR_TEMP_1070 to LOCATION_ALREADY_EXISTS

  • [SPARK-37980] [SQL] Extend METADATA column to support row indexes for Parquet files

  • [SPARK-40967] [SC-114448][SQL] Migrate failAnalysis() onto error classes

  • [SPARK-34265] [SC-115486][SC-113788][PYTHON][SQL] Instrument Python UDFs using SQL metrics

  • [SPARK-41012] [SC-114873][SQL] Rename _LEGACY_ERROR_TEMP_1022 to ORDER_BY_POS_OUT_OF_RANGE

  • [SPARK-41042] [SC-115142][SQL] Rename PARSE_CHAR_MISSING_LENGTH to DATATYPE_MISSING_SIZE

  • [SPARK-40752] [SC-114442][SQL] Migrate type check failures of misc expressions onto error classes

  • [SPARK-37945] [SC-115534][SQL][CORE] Use error classes in the execution errors of arithmetic ops

  • [SPARK-40374] [SC-114705][SQL] Migrate type check failures of type creators onto error classes

  • [SPARK-41056] [SC-115354][R] Fix new R_LIBS_SITE behavior introduced in R 4.2

  • [SPARK-41041] [SC-115357][SQL] Integrate _LEGACY_ERROR_TEMP_1279 into TABLE_OR_VIEW_ALREADY_EXISTS

  • [SPARK-40663] [SC-114852][SQL] Migrate execution errors onto error classes: LEGACYERROR_TEMP_2201-2225

  • [SPARK-41020] [SC-114984][SQL] Rename the error class _LEGACY_ERROR_TEMP_1019 to STAR_GROUP_BY_POS

  • [SPARK-41035] [SC-115233][SQL] Don’t patch foldable children of aggregate functions in RewriteDistinctAggregates

  • [SPARK-39778] [SC-114353][SQL] Improve error classes and messages

  • [SPARK-40798] [SC-113782][SQL] Alter partition should verify value follow storeAssignmentPolicy

  • [SPARK-40810] [SC-113533][SQL] Use SparkIllegalArgumentException instead of IllegalArgumentException in CreateDatabaseCommand & AlterDatabaseSetLocationCommand

  • [SPARK-40921] [SC-115140][SC-114585][SQL] Add WHEN NOT MATCHED BY SOURCE clause to MERGE INTO

  • [SPARK-40360] [SC-114965] ALREADY_EXISTS and NOT_FOUND exceptions

  • [SPARK-41007] [SC-115085][SQL] Add missing serializer for java.math.BigInteger

  • [SPARK-40751] [SC-114254][SQL] Migrate type check failures of high order functions onto error classes

  • [SPARK-40248] [SC-114704][SQL] Use larger number of bits to build Bloom filter

  • [SPARK-41040] [SC-115145][SS] Fix self-union streaming query failure when using readStream.table

  • [SPARK-32380] [SC-114966][SQL] Fixing access of HBase table via Hive from Spark

  • [SPARK-40749] [SC-114860][SQL] Migrate type check failures of generators onto error classes

  • [SPARK-40925] [SC-114445][SQL][SS][WARMFIX][12.x] Fix stateful operator late re…

  • [SPARK-40654] [SC-112783] Remove temporary log lines

  • [SPARK-40742] [SC-114913][SC-112646][CORE][SQL] Fix Java compilation warnings related to generic type

  • [SPARK-40657] [SC-113642][Cherry-pick] Add support for Java classes in Protob…

  • [SPARK-40898] [SC-113932][SQL] Quote function names in datatype mismatch errors

  • [SPARK-40760] [SC-113750][SQL] Migrate type check failures of interval expressions onto error classes

  • [SPARK-36114] [SC-113771][SQL] Support subqueries with correlated non-equality predicates

  • [SPARK-40856] [SC-113738][SQL] Update the error template of WRONG_NUM_PARAMS

  • [SPARK-40759] [SC-114251][SQL] Migrate type check failures of time window onto error classes

  • [SPARK-40750] [SC-113812][SQL] Migrate type check failures of math expressions onto error classes

  • [SPARK-40756] [SC-113749][SQL] Migrate type check failures of string expressions onto error classes

  • [SPARK-40768] [SC-113640][SQL] Migrate type check failures of bloom_filter_agg() onto error classes

  • [SPARK-40369] [SC-113402][CORE][SQL] Migrate the type check failures of calls via reflection onto error classes

  • [SPARK-39445] [SQL] Remove the window if windowExpressions is empty in column pruning

  • [SPARK-40761] [SC-113245][SQL] Migrate type check failures of percentile expressions onto error classes

  • [SPARK-40361] [SC-112789][SQL] Migrate arithmetic type check failures onto error classes

  • [SPARK-40714] [SC-112576][SQL] Remove PartitionAlreadyExistsException

  • [SPARK-40702] [SC-112437][SQL] Fix partition specs in PartitionsAlreadyExistException

  • [SPARK-40358] [SC-112631][SQL] Migrate collection type check failures onto error classes

  • [SPARK-40910] [SC-114259][SQL] Replace UnsupportedOperationException with SparkUnsupportedOperationException

  • [SPARK-39876] [SC-112429][SQL] Add UNPIVOT to SQL syntax

  • [SPARK-39783] [SC-113389][SQL] Quote qualifiedName to fix backticks for column candidates in error messages

  • [SPARK-40663] [SC-112520][SQL] Migrate execution errors onto error classes: LEGACYERROR_TEMP_2026-2282

  • [SPARK-40975] [SC-114555][SQL] Rename the error class _LEGACY_ERROR_TEMP_0021 to UNSUPPORTED_TYPED_LITERAL

  • [SPARK-37935] [SC-114472][SQL] Eliminate separate error sub-classes fields

  • [SPARK-40944] [SC-114530][SQL] Relax ordering constraint for CREATE TABLE column options

  • [SPARK-40815] [SC-114528][SQL] Add DelegateSymlinkTextInputFormat to workaround SymlinkTextInputSplit bug

  • [SPARK-40933] [SC-114441][SQL] Reimplement df.stat.{cov, corr} with built-in sql functions

  • [SPARK-40932] [SC-114439][CORE] Fix issue messages for allGather are overridden

  • [SPARK-39312] [SQL] Use parquet native In predicate for in filter push down

  • [SPARK-40862] [SC-114352][SQL] Support non-aggregated subqueries in RewriteCorrelatedScalarSubquery

  • [SPARK-40963] [SC-114446][SQL] Set nullable correctly in project created by ExtractGenerator

  • [SPARK-40663] [SC-112295][SQL] Migrate execution errors onto error classes: LEGACYERROR_TEMP_2000-2024

  • [SPARK-40892] [SC-114347][SQL][SS] Loosen the requirement of window_time rule - allow multiple window_time calls

  • [SPARK-40540] [SC-111639][SQL] Migrate compilation errors onto error classes: LEGACYERROR_TEMP_1100-1347

  • [SPARK-40924] [SC-114258][SQL] Fix for Unhex when input has odd number of symbols

  • [SPARK-40821] [SC-113754][SQL][CORE][PYTHON][SS] Introduce window_time function to extract event time from the window column

  • [SPARK-40800] [SC-113772][SQL] Always inline expressions in OptimizeOneRowRelationSubquery

  • [SPARK-40900] [SC-114006][SQL] Reimplement frequentItems with dataframe operations

  • [SPARK-40735] [SC-114022][SC-112657] Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable

  • [SPARK-40773] [SC-113091][SQL] Refactor checkCorrelationsInSubquery

  • [SPARK-40615] [SC-113400][SQL] Check unsupported data types when decorrelating subqueries

  • [SPARK-40660] [SC-112293][CORE][SQL] Switch to XORShiftRandom to distribute elements

  • [SPARK-40540] [SC-113926][SC-111331][SQL] Migrate compilation errors onto error classes

  • [SPARK-40551] [SC-113135][SQL] DataSource V2: Add APIs for delta-based row-level operations

  • [SPARK-39391] [SC-110676][CORE] Reuse Partitioner classes

  • [SPARK-40368] [SC-113396][SQL] Migrate Bloom Filter type check failures onto error classes

  • [SPARK-39146] [SC-111814][CORE][SQL] Introduce local singleton for ObjectMapper that may be reused

  • [SPARK-40357] [SC-113786][SC-111352][SQL] Migrate window type check failures onto error classes

  • [SPARK-40874] [SC-113756][PYTHON] Fix broadcasts in Python UDFs when encryption enabled

  • [SPARK-40359] [SC-113770][SC-111146][SQL] Migrate type check fails in CSV/JSON expressions to error classes

  • [SPARK-40880] [SC-113764][SQL] Reimplement summary with dataframe operations

  • [SPARK-40877] [SC-113763][SQL] Reimplement crosstab with dataframe operations

  • [SPARK-40382] [SC-113098][SQL] Group distinct aggregate expressions by semantically equivalent children in RewriteDistinctAggregates

  • [SPARK-40826] [SC-113532][SS] Add additional checkpoint rename file check

  • [SPARK-40829] [SC-113426][SQL] STORED AS serde in CREATE TABLE LIKE view does not work

  • [SPARK-40844] [SC-113542][SS] Flip the default value of Kafka offset fetching config

  • [SPARK-40488] [SC-113568][SC-111160] Do not wrap exceptions thrown when datasource write fails

  • [SPARK-40560] [SC-113631][SC-111335][SQL] Rename message to messageTemplate in the STANDARD format of errors

  • [SPARK-40618] [SC-113096][SQL] Fix bug in MergeScalarSubqueries rule with nested subqueries using reference tracking

  • [SPARK-40530] [SC-113528][SC-111332][SQL] Add error-related developer APIs

  • [SPARK-40806] [SC-113250][SQL] Typo fix: CREATE TABLE -> REPLACE TABLE

  • [SPARK-40646] [SC-113379][SQL] Fix returning partial results in JSON data source and JSON functions

  • [SPARK-40654] [Cherry-pick][SC-112783][SQL] Protobuf support for Spark - from…

  • [SPARK-40753] [SC-113539][SQL] Fix bug in test case for catalog directory operation

  • [SPARK-40765] [SC-113056][SQL] Optimize redundant fs operation in CommandUtils#calculateSingleLocationSize#getPathSize method

  • [SPARK-40114] [ES-479282][R][11.X] Arrow 9.0.0 support with SparkR

  • [SPARK-40479] [SC-110935][SQL] Migrate unexpected input type error to an error class

  • [SPARK-40473] [SC-113388][SC-111141][SQL] Migrate parsing errors onto error classes

  • [SPARK-39853] [SC-111867][CORE] Support stage level task resource profile for standalone cluster when dynamic allocation disabled(back port from PR-47157)

  • [SPARK-40407] [SC-111161][SQL] Fix the potential data skew caused by df.repartition

  • [SPARK-40425] [SC-110840][SQL] DROP TABLE does not need to do table lookup

  • [SPARK-40703] [SC-113242][SQL] Introduce shuffle on SinglePartition to improve parallelism

  • [SPARK-40640] [SC-113138][CORE] SparkHadoopUtil to set origin of hadoop/hive config options

  • [SPARK-39062] [11.x][CORE] Add stage level resource scheduling support for standalone cluster

  • [SPARK-33861] [SC-109877]Revert “[SQL] Simplify conditional in predicate”

  • [SPARK-40667] [SC-113136][SQL] Refactor File Data Source Options

  • [SPARK-40370] [SC-110677][SQL] Migrate type check fails to error classes in CAST

  • [SPARK-40611] [SC-113105][SQL] Improve the performance of setInterval & getInterval for UnsafeRow

  • [SPARK-40733] [SC-113126][SQL] Make the contents of SERDEPROPERTIES in the result of ShowCreateTableAsSerdeCommand have a fixed order

  • [SPARK-40585] [SC-112180][SQL] Support double quoted identifiers

  • [SPARK-40772] [SC-112992][SQL] Improve spark.sql.adaptive.skewJoin.skewedPartitionFactor to support Double values

  • [SPARK-40565] [SC-112417][SQL] Don’t push non-deterministic filters to V2 file sources

  • [SPARK-8731] [SC-112784] Beeline doesn’t work with -e option when started in background

  • [SPARK-35242] [SC-111398][SQL] Support changing session catalog’s default database

  • [SPARK-40426] [SC-110672][SQL] Return a map from SparkThrowable.getMessageParameters

  • [SPARK-40494] [SC-112655][SC-111025][CORE][SQL][ML][MLLIB] Optimize the performance of keys.zipWithIndex.toMap code pattern

  • [SPARK-40521] [SC-112430][SQL] Return only exists partitions in PartitionsAlreadyExistException from Hive’s create partition

  • [SPARK-40705] [SC-112567][SQL] Handle case of using mutable array when converting Row to JSON for Scala 2.13

  • [SPARK-39895] [SC-112590][SQL][PYTHON] Support multiple column drop

  • [SPARK-40607] [SC-112296][CORE][SQL][MLLIB][SS] Remove redundant string interpolator operations

  • [SPARK-40420] [SC-112604][SC-110569][SQL] Sort error message parameters by names in the JSON formats

  • [SPARK-40403] [SC-110564][SQL] Calculate unsafe array size using longs to avoid negative size in error message

  • [SPARK-40482] [SC-110831][SQL] Revert SPARK-24544 Print actual failure cause when look up function failed

  • [SPARK-40400] [SC-112555][SC-110397][SQL] Pass error message parameters to exceptions as maps

  • [SPARK-40628] [SC-112414][SQL] Do not push complex left semi/anti join condition through project

  • [SPARK-40562] [SC-111455][SQL] Add spark.sql.legacy.groupingIdWithAppendedUserGroupBy

  • [SPARK-40501] [SC-111333][SQL] Add PushProjectionThroughLimit for Optimizer

  • [SPARK-39200] [SC-111244][CORE] Make Fallback Storage readFully on content

  • [SPARK-38717] [SC-111241][SQL] Handle Hive’s bucket spec case preserving behavior

  • [SPARK-40385] [SC-111123][SQL] Fix interpreted path for companion object constructor

  • [SPARK-40216] [SC-112413][SQL] Extract common ParquetUtils.prepareWrite method to deduplicate code in ParquetFileFormat and ParquetWrite

  • [SPARK-40636] [SC-112160][CORE] Fix wrong remained shuffles log in BlockManagerDecommissioner

  • [SPARK-40617] [SC-112009] Fix race condition at the handling of ExecutorMetricsPoller’s stageTCMP entries

  • [SPARK-40618] [SC-112046][SQL] Fix bug in MergeScalarSubqueries rule with nested subqueries

  • [SPARK-40612] [SC-111933][CORE] Fixing the principal used for delegation token renewal on non-YARN resource managers

  • [SPARK-40595] [SC-111662][SQL] Improve error message for unused CTE relations

  • [SPARK-40314] [SC-111879][SQL][PYTHON] Add scala and python bindings for inline and inline_outer

  • [SPARK-40416] [SC-110945][SQL] Move subquery expression CheckAnalysis error messages to use the new error framework

  • [SPARK-40318] [SC-111865][SC-109986][SQL] try_avg() should throw the exceptions from its child

  • [SPARK-40509] [SC-111644][SS][PYTHON] Add example for applyInPandasWithState

  • [SPARK-40310] [SC-111329][SC-109842][SQL] try_sum() should throw the exceptions from its child

  • [SPARK-40016] [SQL] Remove unnecessary TryEval in the implementation of try_sum()

  • [SPARK-40527] [SC-111224][SQL] Keep struct field names or map keys in CreateStruct

  • [SPARK-38098] [SC-111178][PYTHON] Add support for ArrayType of nested StructType to arrow-based conversion

  • [SPARK-40324] [SC-111359][SC-110293][SQL] Provide query context in AnalysisException

  • [SPARK-40492] [SC-111324][SS] Do maintenance before streaming StateStore unload

  • [SPARK-40487] [SC-111124][SQL] Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

  • [SPARK-40474] [SC-111273][SQL] Correct CSV schema inference and data parsing behavior on columns with mixed dates and timestamps

  • [SPARK-40508] [SC-111066][SQL] Treat unknown partitioning as UnknownPartitioning

  • [SPARK-40496] [SC-111234][SC-111013][SQL] Fix configs to control “enableDateTimeParsingFallback”

  • [SPARK-40435] [SC-111214][SS][PYTHON] Add test suites for applyInPandasWithState in PySpark

  • [SPARK-40434] [SC-111125][SS][PYTHON] Implement applyInPandasWithState in PySpark

System environment

  • Operating System: Ubuntu 20.04.5 LTS

  • Java: Zulu 8.64.0.19-CA-linux64

  • Scala: 2.12.14

  • Python: 3.9.5

  • R: 4.2.2

  • Delta Lake: 2.2.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

asttokens

2.0.5

attrs

21.4.0

backcall

0.2.0

backports.entry-points-selectable

1.2.0

beautifulsoup4

4.11.1

black

22.3.0

bleach

4.1.0

boto3

1.21.32

botocore

1.24.32

certifi

2021.10.8

cffi

1.15.0

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

cryptography

3.4.8

cycler

0.11.0

Cython

0.29.28

dbus-python

1.2.16

debugpy

1.5.1

decorator

5.1.1

defusedxml

0.7.1

distlib

0.3.6

entrypoints

0.4

executing

0.8.3

facets-overview

1.0.0

fastjsonschema

2.16.2

filelock

3.8.0

fonttools

4.25.0

idna

3.3

ipykernel

6.15.3

ipython

8.5.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

jedi

0.18.1

Jinja2

2.11.3

jmespath

0.10.0

joblib

1.1.0

jsonschema

4.4.0

jupyter-client

6.1.12

jupyter_core

4.11.2

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

kiwisolver

1.3.2

MarkupSafe

2.0.1

matplotlib

3.5.1

matplotlib-inline

0.1.2

mistune

0.8.4

mypy-extensions

0.4.3

nbclient

0.5.13

nbconvert

6.4.4

nbformat

5.3.0

nest-asyncio

1.5.5

notebook

6.4.8

numpy

1.21.5

packaging

21.3

pandas

1.4.2

pandocfilters

1.5.0

parso

0.8.3

pathspec

0.9.0

patsy

0.5.2

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.0.1

pip

21.2.4

platformdirs

2.5.4

plotly

5.6.0

prometheus-client

0.13.1

prompt-toolkit

3.0.20

protobuf

3.19.4

psutil

5.8.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

7.0.0

pycparser

2.21

Pygments

2.11.2

PyGObject

3.36.0

pyodbc

4.0.32

pyparsing

3.0.4

pyrsistent

0.18.0

python-dateutil

2.8.2

pytz

2021.3

pyzmq

22.3.0

requests

2.27.1

requests-unixsocket

0.2.0

s3transfer

0.5.0

scikit-learn

1.0.2

scipy

1.7.3

seaborn

0.11.2

Send2Trash

1.8.0

setuptools

58.0.4

six

1.16.0

soupsieve

2.3.1

ssh-import-id

5.10

stack-data

0.2.0

statsmodels

0.13.2

tenacity

8.0.1

terminado

0.13.1

testpath

0.5.0

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tomli

1.2.2

tornado

6.1

traitlets

5.1.1

typing_extensions

4.1.1

unattended-upgrades

0.1

urllib3

1.26.9

virtualenv

20.8.0

wcwidth

0.2.5

webencodings

0.5.1

wheel

0.37.0

widgetsnbextension

3.6.1

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-11-11.

Library

Version

Library

Version

Library

Version

arrow

10.0.0

askpass

1.1

assertthat

0.2.1

backports

1.4.1

base

4.2.2

base64enc

0.1-3

bit

4.0.4

bit64

4.0.5

blob

1.2.3

boot

1.3-28

brew

1.0-8

brio

1.1.3

broom

1.0.1

bslib

0.4.1

cachem

1.0.6

callr

3.7.3

caret

6.0-93

cellranger

1.1.0

chron

2.3-58

class

7.3-20

cli

3.4.1

clipr

0.8.0

clock

0.6.1

cluster

2.1.4

codetools

0.2-18

colorspace

2.0-3

commonmark

1.8.1

compiler

4.2.2

config

0.3.1

cpp11

0.4.3

crayon

1.5.2

credentials

1.3.2

curl

4.3.3

data.table

1.14.4

datasets

4.2.2

DBI

1.1.3

dbplyr

2.2.1

desc

1.4.2

devtools

2.4.5

diffobj

0.3.5

digest

0.6.30

downlit

0.4.2

dplyr

1.0.10

dtplyr

1.2.2

e1071

1.7-12

ellipsis

0.3.2

evaluate

0.18

fansi

1.0.3

farver

2.1.1

fastmap

1.1.0

fontawesome

0.4.0

forcats

0.5.2

foreach

1.5.2

foreign

0.8-82

forge

0.2.0

fs

1.5.2

future

1.29.0

future.apply

1.10.0

gargle

1.2.1

generics

0.1.3

gert

1.9.1

ggplot2

3.4.0

gh

1.3.1

gitcreds

0.1.2

glmnet

4.1-4

globals

0.16.1

glue

1.6.2

googledrive

2.0.0

googlesheets4

1.0.1

gower

1.0.0

graphics

4.2.2

grDevices

4.2.2

grid

4.2.2

gridExtra

2.3

gsubfn

0.7

gtable

0.3.1

hardhat

1.2.0

haven

2.5.1

highr

0.9

hms

1.1.2

htmltools

0.5.3

htmlwidgets

1.5.4

httpuv

1.6.6

httr

1.4.4

ids

1.0.1

ini

0.3.1

ipred

0.9-13

isoband

0.2.6

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.3

KernSmooth

2.23-20

knitr

1.40

labeling

0.4.2

later

1.3.0

lattice

0.20-45

lava

1.7.0

lifecycle

1.0.3

listenv

0.8.0

lubridate

1.9.0

magrittr

2.0.3

markdown

1.3

MASS

7.3-58

Matrix

1.5-1

memoise

2.0.1

methods

4.2.2

mgcv

1.8-41

mime

0.12

miniUI

0.1.1.1

ModelMetrics

1.2.2.2

modelr

0.1.9

munsell

0.5.0

nlme

3.1-160

nnet

7.3-18

numDeriv

2016.8-1.1

openssl

2.0.4

parallel

4.2.2

parallelly

1.32.1

pillar

1.8.1

pkgbuild

1.3.1

pkgconfig

2.0.3

pkgdown

2.0.6

pkgload

1.3.1

plogr

0.2.0

plyr

1.8.7

praise

1.0.0

prettyunits

1.1.1

pROC

1.18.0

processx

3.8.0

prodlim

2019.11.13

profvis

0.3.7

progress

1.2.2

progressr

0.11.0

promises

1.2.0.1

proto

1.0.0

proxy

0.4-27

ps

1.7.2

purrr

0.3.5

r2d3

0.2.6

R6

2.5.1

ragg

1.2.4

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.9

RcppEigen

0.3.3.9.3

readr

2.1.3

readxl

1.4.1

recipes

1.0.3

rematch

1.0.1

rematch2

2.1.2

remotes

2.4.2

reprex

2.0.2

reshape2

1.4.4

rlang

1.0.6

rmarkdown

2.18

RODBC

1.3-19

roxygen2

7.2.1

rpart

4.1.19

rprojroot

2.0.3

Rserve

1.8-11

RSQLite

2.2.18

rstudioapi

0.14

rversions

2.1.2

rvest

1.0.3

sass

0.4.2

scales

1.2.1

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.7.3

sourcetools

0.1.7

sparklyr

1.7.8

SparkR

3.3.1

spatial

7.3-11

splines

4.2.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.2.2

stats4

4.2.2

stringi

1.7.8

stringr

1.4.1

survival

3.4-0

sys

3.4.1

systemfonts

1.0.4

tcltk

4.2.2

testthat

3.1.5

textshaping

0.3.6

tibble

3.1.8

tidyr

1.2.1

tidyselect

1.2.0

tidyverse

1.3.2

timechange

0.1.1

timeDate

4021.106

tinytex

0.42

tools

4.2.2

tzdb

0.3.0

urlchecker

1.0.1

usethis

2.1.6

utf8

1.2.2

utils

4.2.2

uuid

1.1-0

vctrs

0.5.0

viridisLite

0.4.1

vroom

1.6.0

waldo

0.4.0

whisker

0.4

withr

2.5.0

xfun

0.34

xml2

1.3.3

xopen

1.0.0

xtable

1.8-4

yaml

2.3.6

zip

2.2.2

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.189

com.amazonaws

aws-java-sdk-cloudformation

1.12.189

com.amazonaws

aws-java-sdk-cloudfront

1.12.189

com.amazonaws

aws-java-sdk-cloudhsm

1.12.189

com.amazonaws

aws-java-sdk-cloudsearch

1.12.189

com.amazonaws

aws-java-sdk-cloudtrail

1.12.189

com.amazonaws

aws-java-sdk-cloudwatch

1.12.189

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.189

com.amazonaws

aws-java-sdk-codedeploy

1.12.189

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.189

com.amazonaws

aws-java-sdk-cognitosync

1.12.189

com.amazonaws

aws-java-sdk-config

1.12.189

com.amazonaws

aws-java-sdk-core

1.12.189

com.amazonaws

aws-java-sdk-datapipeline

1.12.189

com.amazonaws

aws-java-sdk-directconnect

1.12.189

com.amazonaws

aws-java-sdk-directory

1.12.189

com.amazonaws

aws-java-sdk-dynamodb

1.12.189

com.amazonaws

aws-java-sdk-ec2

1.12.189

com.amazonaws

aws-java-sdk-ecs

1.12.189

com.amazonaws

aws-java-sdk-efs

1.12.189

com.amazonaws

aws-java-sdk-elasticache

1.12.189

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.189

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.189

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.189

com.amazonaws

aws-java-sdk-emr

1.12.189

com.amazonaws

aws-java-sdk-glacier

1.12.189

com.amazonaws

aws-java-sdk-glue

1.12.189

com.amazonaws

aws-java-sdk-iam

1.12.189

com.amazonaws

aws-java-sdk-importexport

1.12.189

com.amazonaws

aws-java-sdk-kinesis

1.12.189

com.amazonaws

aws-java-sdk-kms

1.12.189

com.amazonaws

aws-java-sdk-lambda

1.12.189

com.amazonaws

aws-java-sdk-logs

1.12.189

com.amazonaws

aws-java-sdk-machinelearning

1.12.189

com.amazonaws

aws-java-sdk-opsworks

1.12.189

com.amazonaws

aws-java-sdk-rds

1.12.189

com.amazonaws

aws-java-sdk-redshift

1.12.189

com.amazonaws

aws-java-sdk-route53

1.12.189

com.amazonaws

aws-java-sdk-s3

1.12.189

com.amazonaws

aws-java-sdk-ses

1.12.189

com.amazonaws

aws-java-sdk-simpledb

1.12.189

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.189

com.amazonaws

aws-java-sdk-sns

1.12.189

com.amazonaws

aws-java-sdk-sqs

1.12.189

com.amazonaws

aws-java-sdk-ssm

1.12.189

com.amazonaws

aws-java-sdk-storagegateway

1.12.189

com.amazonaws

aws-java-sdk-sts

1.12.189

com.amazonaws

aws-java-sdk-support

1.12.189

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.189

com.amazonaws

jmespath-java

1.12.189

com.chuusai

shapeless_2.12

2.3.3

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.13.4

com.fasterxml.jackson.core

jackson-core

2.13.4

com.fasterxml.jackson.core

jackson-databind

2.13.4.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.13.4

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.13.4

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.13.4

com.fasterxml.jackson.module

jackson-module-paranamer

2.13.4

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.13.4

com.github.ben-manes.caffeine

caffeine

2.3.4

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

core

1.1.2

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java-natives

1.1

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java-natives

1.1

com.github.fommil.netlib

netlib-native_ref-linux-x86_64-natives

1.1

com.github.fommil.netlib

netlib-native_system-linux-x86_64-natives

1.1

com.github.luben

zstd-jni

1.5.2-1

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.8.6

com.google.crypto.tink

tink

1.6.1

com.google.flatbuffers

flatbuffers-java

1.12.0

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.h2database

h2

2.0.204

com.helger

profiler

1.1.1

com.jcraft

jsch

0.1.50

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.ning

compress-lzf

1.1

com.sun.mail

javax.mail

1.5.2

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.2.1

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.0

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.15

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.3.3

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.11.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

2.2.1

dev.ludovic.netlib

blas

2.2.1

dev.ludovic.netlib

lapack

2.2.1

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.21

io.delta

delta-sharing-spark_2.12

0.5.2

io.dropwizard.metrics

metrics-core

4.1.1

io.dropwizard.metrics

metrics-graphite

4.1.1

io.dropwizard.metrics

metrics-healthchecks

4.1.1

io.dropwizard.metrics

metrics-jetty9

4.1.1

io.dropwizard.metrics

metrics-jmx

4.1.1

io.dropwizard.metrics

metrics-json

4.1.1

io.dropwizard.metrics

metrics-jvm

4.1.1

io.dropwizard.metrics

metrics-servlets

4.1.1

io.netty

netty-all

4.1.74.Final

io.netty

netty-buffer

4.1.74.Final

io.netty

netty-codec

4.1.74.Final

io.netty

netty-common

4.1.74.Final

io.netty

netty-handler

4.1.74.Final

io.netty

netty-resolver

4.1.74.Final

io.netty

netty-tcnative-classes

2.0.48.Final

io.netty

netty-transport

4.1.74.Final

io.netty

netty-transport-classes-epoll

4.1.74.Final

io.netty

netty-transport-classes-kqueue

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-aarch_64

4.1.74.Final

io.netty

netty-transport-native-epoll-linux-x86_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-aarch_64

4.1.74.Final

io.netty

netty-transport-native-kqueue-osx-x86_64

4.1.74.Final

io.netty

netty-transport-native-unix-common

4.1.74.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.10.13

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.2

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.snowflake

snowflake-jdbc

3.13.22

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.8

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.9.2

org.apache.ant

ant-jsch

1.9.2

org.apache.ant

ant-launcher

1.9.2

org.apache.arrow

arrow-format

7.0.0

org.apache.arrow

arrow-memory-core

7.0.0

org.apache.arrow

arrow-memory-netty

7.0.0

org.apache.arrow

arrow-vector

7.0.0

org.apache.avro

avro

1.11.0

org.apache.avro

avro-ipc

1.11.0

org.apache.avro

avro-mapred

1.11.0

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.21

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-api

3.3.4-databricks

org.apache.hadoop

hadoop-client-runtime

3.3.4

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.7.2

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.13

org.apache.httpcomponents

httpcore

4.4.14

org.apache.ivy

ivy

2.5.0

org.apache.logging.log4j

log4j-1.2-api

2.18.0

org.apache.logging.log4j

log4j-api

2.18.0

org.apache.logging.log4j

log4j-core

2.18.0

org.apache.logging.log4j

log4j-slf4j-impl

2.18.0

org.apache.mesos

mesos-shaded-protobuf

1.4.0

org.apache.orc

orc-core

1.7.6

org.apache.orc

orc-mapreduce

1.7.6

org.apache.orc

orc-shims

1.7.6

org.apache.parquet

parquet-column

1.12.3-databricks-0001

org.apache.parquet

parquet-common

1.12.3-databricks-0001

org.apache.parquet

parquet-encoding

1.12.3-databricks-0001

org.apache.parquet

parquet-format-structures

1.12.3-databricks-0001

org.apache.parquet

parquet-hadoop

1.12.3-databricks-0001

org.apache.parquet

parquet-jackson

1.12.3-databricks-0001

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.xbean

xbean-asm9-shaded

4.20

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.2

org.apache.zookeeper

zookeeper-jute

3.6.2

org.checkerframework

checker-qual

3.5.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-client

9.4.46.v20220331

org.eclipse.jetty

jetty-continuation

9.4.46.v20220331

org.eclipse.jetty

jetty-http

9.4.46.v20220331

org.eclipse.jetty

jetty-io

9.4.46.v20220331

org.eclipse.jetty

jetty-jndi

9.4.46.v20220331

org.eclipse.jetty

jetty-plus

9.4.46.v20220331

org.eclipse.jetty

jetty-proxy

9.4.46.v20220331

org.eclipse.jetty

jetty-security

9.4.46.v20220331

org.eclipse.jetty

jetty-server

9.4.46.v20220331

org.eclipse.jetty

jetty-servlet

9.4.46.v20220331

org.eclipse.jetty

jetty-servlets

9.4.46.v20220331

org.eclipse.jetty

jetty-util

9.4.46.v20220331

org.eclipse.jetty

jetty-util-ajax

9.4.46.v20220331

org.eclipse.jetty

jetty-webapp

9.4.46.v20220331

org.eclipse.jetty

jetty-xml

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-api

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-client

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-common

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-server

9.4.46.v20220331

org.eclipse.jetty.websocket

websocket-servlet

9.4.46.v20220331

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.36

org.glassfish.jersey.containers

jersey-container-servlet-core

2.36

org.glassfish.jersey.core

jersey-client

2.36

org.glassfish.jersey.core

jersey-common

2.36

org.glassfish.jersey.core

jersey-server

2.36

org.glassfish.jersey.inject

jersey-hk2

2.36

org.hibernate.validator

hibernate-validator

6.1.0.Final

org.javassist

javassist

3.25.0-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mariadb.jdbc

mariadb-java-client

2.7.4

org.mlflow

mlflow-spark

1.27.0

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.3.3

org.roaringbitmap

RoaringBitmap

0.9.25

org.roaringbitmap

shims

0.9.25

org.rocksdb

rocksdbjni

6.24.2

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.14

org.scala-lang

scala-library_2.12

2.12.14

org.scala-lang

scala-reflect_2.12

2.12.14

org.scala-lang.modules

scala-collection-compat_2.12

2.4.3

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.0.8

org.scalanlp

breeze-macros_2.12

1.2

org.scalanlp

breeze_2.12

1.2

org.scalatest

scalatest_2.12

3.0.8

org.slf4j

jcl-over-slf4j

1.7.36

org.slf4j

jul-to-slf4j

1.7.36

org.slf4j

slf4j-api

1.7.36

org.spark-project.spark

unused

1.0.0

org.threeten

threeten-extra

1.5.0

org.tukaani

xz

1.8

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

macro-compat_2.12

1.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.0.7.Final

org.xerial

sqlite-jdbc

3.8.11.2

org.xerial.snappy

snappy-java

1.1.8.4

org.yaml

snakeyaml

1.24

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1