Databricks Runtime 15.4 LTS

The following release notes provide information about Databricks Runtime 15.4 LTS, powered by Apache Spark 3.5.0.

Databricks released this version in August 2024.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

Tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

Behavioral changes

Using a VARIANT as the input or output type with a Python UDF, UDAF, or UDTF throws an exception

[Breaking change] In Databricks Runtime 15.3 and above, calling any Python user-defined function (UDF), user-defined aggregate function (UDAF), or user-defined table function (UDTF) that uses a VARIANT type as an argument or return value throws an exception. This change is made to prevent issues that might occur because of an invalid value returned by one of these functions. To learn more about the VARIANT type, see use VARIANTs to store semi-structured data.

Change to the default schema binding mode for views

Views now adapt to schema changes in the underlying query by using schema compensation with regular casting rules. This is a change from the previous default of BINDING mode, which raised errors when a safe cast could not be performed when referencing the view.

See CREATE VIEW and cast function.

Disallow using the undocumented ! syntax instead of NOT outside boolean expressions

With this release, the use of ! as a synonym for NOT outside of boolean expressions is no longer allowed. For example, statements such as the following: CREATE ... IF ! EXISTS, IS ! NULL, a ! NULL column or field property, ! IN and ! BETWEEN, must be replaced with: CREATE ... IF NOT EXISTS, IS NOT NULL, a NOT NULL column or field property, NOT IN and NOT BETWEEN.

This change ensures consistency, aligns with the SQL standard, and makes your SQL more portable.

The boolean prefix operator ! (for example, !is_mgr or !(true AND false)) is unaffected by this change.

Disallow undocumented column definition syntax in views

Databricks supports CREATE VIEW with named columns and column comments. Previously, the specification of column types, NOT NULL constraints, or DEFAULT has been allowed. With this release, you can no longer use this syntax.

This change ensures consistency, aligns with the SQL standard, and supports future enhancements.

Consistent error handling for Base64 decoding in Spark and Photon

This release changes how Photon handles Base64 decoding errors to match the Spark handling of these errors. Before these changes, the Photon and Spark code generation path sometimes failed to raise parsing exceptions, while the Spark interpreted execution correctly raised IllegalArgumentException or ConversionInvalidInputError. This update ensures that Photon consistently raises the same exceptions as Spark during Base64 decoding errors, providing more predictable and reliable error handling.

Adding a CHECK constraint on an invalid column now returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class

To provide more useful error messaging, in Databricks Runtime 15.3 and above, an ALTER TABLE ADD CONSTRAINT statement that includes a CHECK constraint referencing an invalid column name returns the UNRESOLVED_COLUMN.WITH_SUGGESTION error class. Previously, an INTERNAL_ERROR was returned.

New features and improvements

UTF-8 validation functions

This release introduces the following functions for validating UTF-8 strings:

  • is_valid_utf8 verified whether a string is a valid UTF-8 string.

  • make_valid_utf8 converts a potentially invalid UTF-8 string to a valid UTF-8 string using substitution characters

  • validate_utf8 raises an error if the input is not a valid UTF-8 string.

  • try_validate_utf8 returns NULL if the input is not a valid UTF-8 string.

to_avro and from_avro functions

The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.

Typed Dataset APIs with Scala UDFs

This release includes added support for typed Dataset APIs with Scala user-defined functions (excluding user-defined aggregate functions) on Unity Catalog-enabled compute with shared access mode. See Typed Dataset APIs.

Enable UniForm Iceberg using ALTER TABLE

You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable by altering an existing table.

try_url_decode function

This release introduces the try_url_decode function, which decodes a URL-encoded string. If the string is not in the correct format, the function returns NULL instead of raising an error.

Optionally allow the optimizer to rely on unenforced foreign key constraints

To improve query performance, you can now specify the RELY keyword on FOREIGN KEY constraints when you CREATE or ALTER a table.

Parallelized job runs for selective overwrites

Selective overwrites using replaceWhere now run jobs that delete data and insert new data in parallel, improving query performance and cluster utilization.

Improved performance for change data feed with selective overwrites

Selective overwrites using replaceWhere on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type column present in the underlying Parquet data files to record changes without write amplification.

Improved query latency for the COPY INTO command

This release includes a change that improves the query latency for the COPY INTO command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.

Support for dropping the check constraints table feature

You can now drop the checkConstraints table feature from a Delta table using ALTER TABLE table_name DROP FEATURE checkConstraints. See Disable check constraints.

Single user compute supports fine-grained access control, materialized views, and streaming tables (Public Preview)

When a workspace is enabled for serverless compute, Databricks Runtime 15.4 LTS adds support for fine-grained access control on single user compute. When a query accesses any of the following objects, the single user compute resource on Databricks Runtime 15.4 LTS passes the query to the serverless compute to run data filtering:

  • Views defined over tables on which the user does not have the SELECT privilege.

  • Dynamic views.

  • Tables with row filters or column masks applied.

  • Materialized views and streaming tables.

These queries are unsupported on single user compute running Databricks Runtime 15.3 and below.

For more information, see Fine-grained access control on single user compute.

Expanded support for Java and Scala libraries

Starting in Databricks Runtime 15.4 LTS, all Java and Scala libraries bundled with Databricks Runtime are available in all compute access modes when you use Unity Catalog. To learn more about language support on Unity Catalog-enabled compute, see Compute access mode limitations for Unity Catalog.

Expanded support for Scala Dataset operations

With this release, Unity Catalog-enabled compute resources that use shared access mode support the following Scala Dataset operations: map, mapPartitions, foreachPartition, flatMap, reduce and filter.

Scala is GA on Unity Catalog shared compute

With this release, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs). Structured Streaming, Hive UDFs, and Hive user-defined aggregate functions are not supported. For a complete list of limitations, see Compute access mode limitations for Unity Catalog.

Unity Catalog-governed access to external cloud services using service credentials (Public Preview)

Service credentials enable simple and secure authentication with your cloud tenant’s services using IAM roles and Unity Catalog. See Manage access to external cloud services using service credentials.

Bug fixes

Library upgrades

  • Upgraded Python libraries:

    • azure-core from 1.30.1 to 1.30.2

    • google-auth from 2.29.0 to 2.31.0

    • google-cloud-storage from 2.16.0 to 2.17.0

    • google-resumable-media from 2.7.0 to 2.7.1

    • googleapis-common-protos from 1.63.0 to 1.63.2

    • mlflow-skinny from 2.11.3 to 2.11.4

    • proto-plus from 1.23.0 to 1.24.0

    • s3transfer from 0.10.1 to 0.10.2

  • Upgraded R libraries:

  • Upgraded Java libraries:

    • com.databricks.databricks-sdk-java from 0.17.1 to 0.27.0

    • com.ibm.icu.icu4j from 72.1 to 75.1

    • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.1-linux-x86_64 to 1.6.2-linux-x86_64

Apache Spark

Databricks Runtime 15.4 LTS includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.3, and the following additional bug fixes and improvements made to Spark:

  • [SPARK-48503] [DBRRM-1150][SC-172196][SQL] Allow grouping on expressions in scalar subqueries, if they are bound to outer rows

  • [SPARK-48834] [BEHAVE-79][SC-170972][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation

  • [SPARK-48441] [SC-170980][SQL][WARMFIX] Fix StringTrim behaviour for non-UTF8_BINARY collations

  • [SPARK-48440] [SC-170895][SQL][WARMFIX] Fix StringTranslate behaviour for non-UTF8_BINARY collations

  • [SPARK-48872] [SC-170866][PYTHON] Reduce the overhead of capturecall_site

  • [SPARK-48862] [SC-170845][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled

  • [SPARK-48852] [SC-170837][CONNECT] Fix string trim function in connect

  • [SPARK-48791] [SC-170658][CORE] Fix perf regression caused by the accumulators registration overhead using CopyOnWriteArrayList

  • [SPARK-48118] [SQL] Support SPARK_SQL_LEGACY_CREATE_HIVE_TABLE env variable

  • [SPARK-48241] [SC-165811][SQL] CSV parsing failure with char/varchar type columns

  • [SPARK-48168] [SC-166900][SQL] Add bitwise shifting operators support

  • [SPARK-48148] [SC-165630][CORE] JSON objects should not be modified when read as STRING

  • [SPARK-46625] [SC-170561] CTE with Identifier clause as reference

  • [SPARK-48771] [SC-170546][SQL] Speed up LogicalPlanIntegrity.validateExprIdUniqueness for large query plans

  • [SPARK-48831] [BEHAVE-76][SC-170554][CONNECT] Make default column name of cast compatible with Spark Classic

  • [SPARK-48623] [SC-170544][CORE] Structured logging migrations [Part 2]

  • [SPARK-48296] [SC-166138][SQL] Codegen Support for to_xml

  • [SPARK-48027] [SC-165154][SQL] InjectRuntimeFilter for multi-level join should check child join type

  • [SPARK-48686] [SC-170365][SQL] Improve performance of ParserUtils.unescapeSQLString

  • [SPARK-48798] [SC-170588][PYTHON] Introduce spark.profile.render for SparkSession-based profiling

  • [SPARK-48048] [SC-169099] Revert “[SC-164846][CONNECT][SS] Added client side listener support for Scala”

  • [SPARK-47910] [SC-168929][CORE] close stream when DiskBlockObjectWriter closeResources to avoid memory leak

  • [SPARK-48816] [SC-170547][SQL] Shorthand for interval converters in UnivocityParser

  • [SPARK-48589] [SC-170132][SQL][SS] Add option snapshotStartBatchId and snapshotPartitionId to state data source

  • [SPARK-48280] [SC-170293][SQL] Improve collation testing surface area using expression walking

  • [SPARK-48837] [SC-170540][ML] In CountVectorizer, only read binary parameter once per transform, not once per row

  • [SPARK-48803] [SC-170541][SQL] Throw internal error in Orc(De)serializer to align with ParquetWriteSupport

  • [SPARK-48764] [SC-170129][PYTHON] Filtering out IPython-related frames from user stack

  • [SPARK-48818] [SC-170414][PYTHON] Simplify percentile functions

  • [SPARK-48479] [SC-169079][SQL] Support creating scalar and table SQL UDFs in parser

  • [SPARK-48697] [SC-170122][LC-4703][SQL] Add collation aware string filters

  • [SPARK-48800] [SC-170409][CONNECT][SS] Deflake ClientStreamingQuerySuite

  • [SPARK-48738] [SC-169814][SQL] Correct since version for built-in func alias random, position, mod, cardinality, current_schema, user, session_user,char_length,character_length

  • [SPARK-48638] [SC-169575][CONNECT] Add ExecutionInfo support for DataFrame

  • [SPARK-48064] [SC-164697][SQL] Update error messages for routine related error classes

  • [SPARK-48810] [CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server

  • [SPARK-48650] [15.x][PYTHON] Display correct call site from IPython Notebook

  • [SPARK-48271] [SC-166076][SQL] Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER

  • [SPARK-48709] [SC-169603][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS

  • [SPARK-48792] [SC-170335][SQL] Fix regression for INSERT with partial column list to a table with char/varchar

  • [SPARK-48767] [SC-170330][SQL] Fix some error prompts when variant type data is invalid

  • [SPARK-48719] [SC-170339][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null

  • [SPARK-48815] [SC-170334][CONNECT] Update environment when stoping connect session

  • [SPARK-48646] [SC-169020][PYTHON] Refine Python data source API docstring and type hints

  • [SPARK-48806] [SC-170310][SQL] Pass actual exception when url_decode fails

  • [SPARK-47777] [SC-168818] fix python streaming data source connect test

  • [SPARK-48732] [SC-169793][SQL] Cleanup deprecated api usage related to JdbcDialect.compileAggregate

  • [SPARK-48675] [SC-169538][SQL] Fix cache table with collated column

  • [SPARK-48623] [SC-169034][CORE] Structured Logging Migrations

  • [SPARK-48655] [SC-169542][SQL] SPJ: Add tests for shuffle skipping for aggregate queries

  • [SPARK-48586] [SC-169808][SS] Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()

  • [SPARK-48799] [Backport][15.x][SC-170283][SS] Refactor versioning for operator metadata read/write and callers

  • [SPARK-48808] [SC-170309][SQL] Fix NPE when connecting thriftserver through Hive 1.2.1 and the result schema is empty

  • [SPARK-48715] [SC-170291][SQL] Integrate UTF8String validation into collation-aware string function implementations

  • [SPARK-48747] [SC-170120][SQL] Add code point iterator to UTF8String

  • [SPARK-48748] [SC-170115][SQL] Cache numChars in UTF8String

  • [SPARK-48744] [SC-169817][Core] Log entry should be constructed only once

  • [SPARK-46122] [SC-164313][SQL] Set spark.sql.legacy.createHiveTableByDefault to false by default

  • [SPARK-48765] [SC-170119][DEPLOY] Enhance default value evaluation for SPARK_IDENT_STRING

  • [SPARK-48759] [SC-170128][SQL] Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4

  • [SPARK-48598] [SC-169484][PYTHON][CONNECT] Propagate cached schema in dataframe operations

  • [SPARK-48766] [SC-170126][PYTHON] Document the behavior difference of extraction between element_at and try_element_at

  • [SPARK-48768] [SC-170124][PYTHON][CONNECT] Should not cache explain

  • [SPARK-48770] [Backport][15.x][SC-170133][SS] Change to read operator metadata once on driver to check if we can find info for numColsPrefixKey used for session window agg queries

  • [SPARK-48656] [SC-169529][CORE] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in CartesianRDD.getPartitions

  • [SPARK-48597] [SC-168817][SQL] Introduce a marker for isStreaming property in text representation of logical plan

  • [SPARK-48472] [SC-169044][SQL] Enable reflect expressions with collated strings

  • [SPARK-48699] [SC-169597][SQL] Refine collation API

  • [SPARK-48682] [SC-169812][SQL][BEHAVE-58] Use ICU in InitCap expression for UTF8_BINARY strings

  • [SPARK-48282] [SC-169813][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringReplace, FindInSet)

  • [SPARK-47353] [SC-169599][SQL] Enable collation support for the Mode expression

  • [SPARK-48320] [SPARK-48490] Sync the latest Logging trait and test cases from OSS Spark

  • [SPARK-48629] [SC-169479] Migrate the residual code to structured logging framework

  • [SPARK-48681] [SC-169469][SQL][BEHAVE-58] Use ICU in Lower/Upper expressions for UTF8_BINARY strings

  • [SPARK-48573] [15.x][SC-169582][SQL] Upgrade ICU version

  • [SPARK-48687] [Backport][15.x][SS] Add change to perform state schema validation and update on driver for stateful queries

  • [SPARK-47579] [15.x][SC-167310][CORE][PART4] Migrate logInfo with variables to structured logging framework

  • [SPARK-48008] [SC-167363][1/2] Support UDAFs in Spark Connect

  • [SPARK-48578] [SC-169505][SQL] add UTF8 string validation related functions

  • [SPARK-48670] [SC-169598][SQL] Providing suggestion as part of error message when invalid collation name is given

  • [SPARK-48059] [SPARK-48145][SPARK-48134][SPARK-48182][SPARK-48209][SPA… …RK-48291] Structured log framework on the java side

  • [SPARK-47599] [15.x][SC-166000][MLLIB] MLLib: Migrate logWarn with variables to structured logging framework

  • [SPARK-48706] [SC-169589][PYTHON] Python UDF in higher order functions should not throw internal error

  • [SPARK-48498] [BEHAVE-38][SC-168060][SQL] Always do char padding in predicates

  • [SPARK-48662] [SC-169533][SQL] Fix StructsToXml expression with collations

  • [SPARK-48482] [SC-167702][PYTHON][15.x] dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args

  • [SPARK-48678] [SC-169463][CORE] Performance optimizations for SparkConf.get(ConfigEntry)

  • [SPARK-48576] [SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE

  • [SPARK-47927] [SC-164123][SQL]: Fix nullability attribute in UDF decoder

  • [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework (new)

  • [SPARK-48695] [SC-169473][PYTHON] TimestampNTZType.fromInternal not use the deprecated methods

  • [SPARK-48431] [SC-167290][LC-4066][SQL] Do not forward predicates on collated columns to file readers

  • [SPARK-47579] Revert “[SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework”

  • [SPARK-47585] [SC-164306][SQL] SQL core: Migrate logInfo with variables to structured logging framework

  • [SPARK-48466] [SC-169042][SQL] Create dedicated node for EmptyRelation in AQE

  • [SPARK-47579] [SC-165297][CORE][PART1] Migrate logInfo with variables to structured logging framework

  • [SPARK-48410] [SC-168320][SQL] Fix InitCap expression for UTF8_BINARY_LCASE & ICU collations

  • [SPARK-48318] [SC-167709][SQL] Enable hash join support for all collations (complex types)

  • [SPARK-48435] [SC-168128][SQL] UNICODE collation should not support binary equality

  • [SPARK-48555] [SC-169041][SQL][PYTHON][CONNECT] Support using Columns as parameters for several functions in pyspark/scala

  • [SPARK-48591] [SC-169081][PYTHON] Add a helper function to simplify Column.py

  • [SPARK-48574] [SC-169043][SQL] Fix support for StructTypes with collations

  • [SPARK-48305] [SC-166390][SQL] Add collation support for CurrentLike expressions

  • [SPARK-48342] [SC-168941][SQL] Introduction of SQL Scripting Parser

  • [SPARK-48649] [SC-169024][SQL] Add “ignoreInvalidPartitionPaths” and “spark.sql.files.ignoreInvalidPartitionPaths” configs to allow ignoring invalid partition paths

  • [SPARK-48000] [SC-167194][SQL] Enable hash join support for all collations (StringType)

  • [SPARK-48459] [SC-168947][CONNECT][PYTHON] Implement DataFrameQueryContext in Spark Connect

  • [SPARK-48602] [SC-168692][SQL] Make csv generator support different output style with spark.sql.binaryOutputStyle

  • [SPARK-48283] [SC-168129][SQL] Modify string comparison for UTF8_BINARY_LCASE

  • [SPARK-48610] [SC-168830][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG

  • [SPARK-48634] [SC-169021][PYTHON][CONNECT] Avoid statically initialize threadpool at ExecutePlanResponseReattachableIterator

  • [SPARK-47911] [SC-164658][SQL] Introduces a universal BinaryFormatter to make binary output consistent

  • [SPARK-48642] [SC-168889][CORE] False SparkOutOfMemoryError caused by killing task on spilling

  • [SPARK-48572] [SC-168844][SQL] Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions

  • [SPARK-48600] [SC-168841][SQL] Fix FrameLessOffsetWindowFunction expressions implicit casting

  • [SPARK-48644] [SC-168933][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex

  • [SPARK-48587] [SC-168824][VARIANT] Avoid storage amplification when accessing a sub-Variant

  • [SPARK-48647] [SC-168936][PYTHON][CONNECT] Refine the error message for YearMonthIntervalType in df.collect

  • [SPARK-48307] [SC-167802][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node

  • [SPARK-48596] [SC-168581][SQL] Perf improvement for calculating hex string for long

  • [SPARK-48621] [SC-168726][SQL] Fix Like simplification in Optimizer for collated strings

  • [SPARK-47148] [SC-164179][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation

  • [SPARK-48584] [SC-168579][SQL] Perf improvement for unescapePathName

  • [SPARK-48281] [SC-167260][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)

  • [SPARK-48577] [SC-168826][SQL] Invalid UTF-8 byte sequence replacement

  • [SPARK-48595] [SC-168580][CORE] Cleanup deprecated api usage related to commons-compress

  • [SPARK-48030] [SC-164303][SQL] SPJ: cache rowOrdering and structType for InternalRowComparableWrapper

  • [SPARK-48004] [SC-164005][SQL] Add WriteFilesExecBase trait for v1 write

  • [SPARK-48551] [SC-168438][SQL] Perf improvement for escapePathName

  • [SPARK-48565] [SC-168437][UI] Fix thread dump display in UI

  • [SPARK-48364] [SC-166782][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings

  • [SPARK-48421] [SC-168689][SQL] SPJ: Add documentation

  • [SPARK-48604] [SC-168698][SQL] Replace deprecated new ArrowType.Decimal(precision, scale) method call

  • [SPARK-46947] [SC-157561][CORE] Delay memory manager initialization until Driver plugin is loaded

  • [SPARK-48411] [SC-168576][SS][PYTHON] Add E2E test for DropDuplicateWithinWatermark

  • [SPARK-48543] [SC-168697][SS] Track state row validation failures using explicit error class

  • [SPARK-48221] [SC-167143][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)

  • [SPARK-47415] [SC-168441][SQL] Add collation support for Levenshtein expression

  • [SPARK-48593] [SC-168719][PYTHON][CONNECT] Fix the string representation of lambda function

  • [SPARK-48622] [SC-168710][SQL] get SQLConf once when resolving column names

  • [SPARK-48594] [SC-168685][PYTHON][CONNECT] Rename parent field to child in ColumnAlias

  • [SPARK-48403] [SC-168319][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations

  • [SPARK-48162] [SC-166062][SQL] Add collation support for MISC expressions

  • [SPARK-48518] [SC-167718][CORE] Make LZF compression be able to run in parallel

  • [SPARK-48474] [SC-167447][CORE] Fix the class name of the log in SparkSubmitArguments & SparkSubmit

  • [SPARK-48012] [SC-168267][SQL] SPJ: Support Transfrom Expressions for One Side Shuffle

  • [SPARK-48552] [SC-168212][SQL] multi-line CSV schema inference should also throw FAILED_READ_FILE

  • [SPARK-48560] [SC-168268][SS][PYTHON] Make StreamingQueryListener.spark settable

  • [SPARK-48569] [SC-168321][SS][CONNECT] Handle edge cases in query.name

  • [SPARK-47260] [SC-167323][SQL] Assign name to error class LEGACYERROR_TEMP_3250

  • [SPARK-48564] [SC-168327][PYTHON][CONNECT] Propagate cached schema in set operations

  • [SPARK-48155] [SC-165910][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec

  • [SPARK-48506] [SC-167720][CORE] Compression codec short names are case insensitive except for event logging

  • [SPARK-48447] [SC-167607][SS] Check state store provider class before invoking the constructor

  • [SPARK-47977] [SC-167650] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

  • [SPARK-48513] [Backport][15.x][SC-168085][SS] Add error class for state schema compatibility and minor refactoring

  • [SPARK-48413] [SC-167669][SQL] ALTER COLUMN with collation

  • [SPARK-48561] [SC-168250][PS][CONNECT] Throw PandasNotImplementedError for unsupported plotting functions

  • [SPARK-48465] [SC-167531][SQL] Avoid no-op empty relation propagation

  • [SPARK-48553] [SC-168166][PYTHON][CONNECT] Cache more properties

  • [SPARK-48540] [SC-168069][CORE] Avoid ivy output loading settings to stdout

  • [SPARK-48535] [SC-168057][SS] Update config docs to indicate possibility of data loss/corruption issue if skip nulls for stream-stream joins config is enabled

  • [SPARK-48536] [SC-168059][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow

  • [SPARK-47873] [SC-163473][SQL] Write collated strings to Hive metastore using the regular string type

  • [SPARK-48461] [SC-167442][SQL] Replace NullPointerExceptions with error class in AssertNotNull expression

  • [SPARK-47833] [SC-163191][SQL][CORE] Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException

  • [SPARK-47898] [SC-163146][SQL] Port HIVE-12270: Add DBTokenStore support to HS2 delegation token

  • [SPARK-47578] [SC-167497][R] Migrate RPackageUtils with variables to structured logging framework

  • [SPARK-47875] [SC-162935][CORE] Remove spark.deploy.recoverySerializer

  • [SPARK-47552] [SC-160880][CORE] Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s if missing

  • [SPARK-47972] [SC-167692][SQL] Restrict CAST expression for collations

  • [SPARK-48430] [SC-167489][SQL] Fix map value extraction when map contains collated strings

  • [SPARK-47318] [SC-162712][CORE][3.5] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices

  • [SPARK-48503] [BEHAVE-29][ES-1135236][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed

  • [SPARK-48508] [SC-167695][CONNECT][PYTHON] Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}

  • [SPARK-23015] [SC-167188][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure

  • [SPARK-45891] [SC-167608]Revert ” Describe shredding scheme for Variant”

  • [SPARK-48391] [SC-167554][CORE]Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class

  • [SPARK-48496] [SC-167600][CORE] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs

  • [SPARK-48476] [SC-167488][SQL] fix NPE error message for null delmiter csv

  • [SPARK-48489] [SC-167598][SQL] Throw a better, user-facing error when reading illegal schema from text Data Source

  • [SPARK-48471] [SC-167324][CORE] Improve documentation and usage guide for history server

  • [SPARK-45891] [SC-167597] Describe shredding scheme for Variant

  • [SPARK-47333] [SC-159043][SQL] Use checkInputDataTypes to check the parameter types of the function to_xml

  • [SPARK-47387] [SC-159310][SQL] Remove some unused error classes

  • [SPARK-48265] [ES-1131440][SQL] Infer window group limit batch should do constant folding

  • [SPARK-47716] [SC-167444][SQL] Avoid view name conflict in SQLQueryTestSuite semantic sort test case

  • [SPARK-48159] [SC-167141][SQL] Extending support for collated strings on datetime expressions

  • [SPARK-48462] [SC-167271][SQL][Tests] Use withSQLConf in tests: Refactor HiveQuerySuite and HiveTableScanSuite

  • [SPARK-48392] [SC-167195][CORE] Also load spark-defaults.conf when provided --properties-file

  • [SPARK-48468] [SC-167417] Add LogicalQueryStage interface in catalyst

  • [SPARK-47578] [SC-164988][CORE] Manual backport for Spark PR #46309: Migrate logWarning with variables to structured logging framework

  • [SPARK-48415] [SC-167321]Revert “[PYTHON] Refactor TypeName to support parameterized datatypes”

  • [SPARK-46544] [SC-151361][SQL] Support v2 DESCRIBE TABLE EXTENDED with table stats

  • [SPARK-48325] [SC-166963][CORE] Always specify messages in ExecutorRunner.killProcess

  • [SPARK-46841] [SC-167139][SQL] Add collation support for ICU locales and collation specifiers

  • [SPARK-47221] [SC-157870][SQL] Uses signatures from CsvParser to AbstractParser

  • [SPARK-47246] [SC-158138][SQL] Replace InternalRow.fromSeq with new GenericInternalRow to save a collection conversion

  • [SPARK-47597] [SC-163932][STREAMING] Manual backport for Spark PR #46192: Streaming: Migrate logInfo with variables to structured logging framework

  • [SPARK-48415] [SC-167130][PYTHON] Refactor TypeName to support parameterized datatypes

  • [SPARK-48434] [SC-167132][PYTHON][CONNECT] Make printSchema use the cached schema

  • [SPARK-48432] [ES-1097114][SQL] Avoid unboxing integers in UnivocityParser

  • [SPARK-47463] [SC-162840][SQL] Use V2Predicate to wrap expression with return type of boolean

  • [SPARK-47781] [SC-162293][SQL] Handle negative scale decimals for JDBC data sources

  • [SPARK-48394] [SC-166966][CORE] Cleanup mapIdToMapIndex on mapoutput unregister

  • [SPARK-47072] [SC-156933][SQL] Fix supported interval formats in error messages

  • [SPARK-47001] [SC-162487][SQL] Pushdown verification in optimizer

  • [SPARK-48335] [SC-166387][PYTHON][CONNECT] Make _parse_datatype_string compatible with Spark Connect

  • [SPARK-48329] [SC-166518][SQL] Enable spark.sql.sources.v2.bucketing.pushPartValues.enabled by default

  • [SPARK-48412] [SC-166898][PYTHON] Refactor data type json parse

  • [SPARK-48215] [SC-166781][SQL] Extending support for collated strings on date_format expression

  • [SPARK-45009] [SC-166873][SQL][FOLLOW UP] Add error class and tests for decorrelation of predicate subqueries in join condition which reference both join child

  • [SPARK-47960] [SC-165295][SS][15.x] Allow chaining other stateful operators after transformWithState operator.

  • [SPARK-48340] [SC-166468][PYTHON] Support TimestampNTZ infer schema miss prefer_timestamp_ntz

  • [SPARK-48157] [SC-165902][SQL] Add collation support for CSV expressions

  • [SPARK-48158] [SC-165652][SQL] Add collation support for XML expressions

  • [SPARK-48160] [SC-166064][SQL] Add collation support for XPATH expressions

  • [SPARK-48229] [SC-165901][SQL] Add collation support for inputFile expressions

  • [SPARK-48367] [SC-166487][CONNECT] Fix lint-scala for scalafmt to detect files to format properly

  • [SPARK-47858] [SC-163095][SPARK-47852][PYTHON][SQL] Refactoring the structure for DataFrame error context

  • [SPARK-48370] [SC-166787][CONNECT] Checkpoint and localCheckpoint in Scala Spark Connect client

  • [SPARK-48247] [SC-166028][PYTHON] Use all values in a dict when inferring MapType schema

  • [SPARK-48395] [SC-166794][PYTHON] Fix StructType.treeString for parameterized types

  • [SPARK-48393] [SC-166784][PYTHON] Move a group of constants to pyspark.util

  • [SPARK-48372] [SC-166776][SPARK-45716][PYTHON] Implement StructType.treeString

  • [SPARK-48258] [SC-166467][PYTHON][CONNECT] Checkpoint and localCheckpoint in Spark Connect

See Databricks Runtime 15.4 LTS maintenance updates.

System environment

  • Operating System: Ubuntu 22.04.4 LTS

  • Java: Zulu 8.78.0.19-CA-linux64

  • Scala: 2.12.18

  • Python: 3.11.0

  • R: 4.3.2

  • Delta Lake: 3.2.0

Installed Python libraries

Library

Version

Library

Version

Library

Version

asttokens

2.0.5

astunparse

1.6.3

azure-core

1.30.2

azure-storage-blob

12.19.1

azure-storage-file-datalake

12.14.0

backcall

0.2.0

black

23.3.0

blinker

1.4

boto3

1.34.39

botocore

1.34.39

cachetools

5.3.3

certifi

2023.7.22

cffi

1.15.1

chardet

4.0.0

charset-normalizer

2.0.4

click

8.0.4

cloudpickle

2.2.1

comm

0.1.2

contourpy

1.0.5

cryptography

41.0.3

cycler

0.11.0

Cython

0.29.32

databricks-sdk

0.20.0

dbus-python

1.2.18

debugpy

1.6.7

decorator

5.1.1

distlib

0.3.8

entrypoints

0.4

executing

0.8.3

facets-overview

1.1.1

filelock

3.13.4

fonttools

4.25.0

gitdb

4.0.11

GitPython

3.1.43

google-api-core

2.18.0

google-auth

2.31.0

google-cloud-core

2.4.1

google-cloud-storage

2.17.0

google-crc32c

1.5.0

google-resumable-media

2.7.1

googleapis-common-protos

1.63.2

grpcio

1.60.0

grpcio-status

1.60.0

httplib2

0.20.2

idna

3.4

importlib-metadata

6.0.0

ipyflow-core

0.0.198

ipykernel

6.25.1

ipython

8.15.0

ipython-genutils

0.2.0

ipywidgets

7.7.2

isodate

0.6.1

jedi

0.18.1

jeepney

0.7.1

jmespath

0.10.0

joblib

1.2.0

jupyter_client

7.4.9

jupyter_core

5.3.0

keyring

23.5.0

kiwisolver

1.4.4

launchpadlib

1.10.16

lazr.restfulclient

0.14.4

lazr.uri

1.0.6

matplotlib

3.7.2

matplotlib-inline

0.1.6

mlflow-skinny

2.11.4

more-itertools

8.10.0

mypy-extensions

0.4.3

nest-asyncio

1.5.6

numpy

1.23.5

oauthlib

3.2.0

packaging

23.2

pandas

1.5.3

parso

0.8.3

pathspec

0.10.3

patsy

0.5.3

pexpect

4.8.0

pickleshare

0.7.5

Pillow

9.4.0

pip

23.2.1

platformdirs

3.10.0

plotly

5.9.0

prompt-toolkit

3.0.36

proto-plus

1.24.0

protobuf

4.24.1

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

pyarrow

14.0.1

pyasn1

0.4.8

pyasn1-modules

0.2.8

pyccolo

0.0.52

pycparser

2.21

pydantic

1.10.6

Pygments

2.15.1

PyGObject

3.42.1

PyJWT

2.3.0

pyodbc

4.0.38

pyparsing

3.0.9

python-dateutil

2.8.2

python-lsp-jsonrpc

1.1.1

pytz

2022.7

PyYAML

6.0

pyzmq

23.2.0

requests

2.31.0

rsa

4.9

s3transfer

0.10.2

scikit-learn

1.3.0

scipy

1.11.1

seaborn

0.12.2

SecretStorage

3.3.1

setuptools

68.0.0

six

1.16.0

smmap

5.0.1

sqlparse

0.5.0

ssh-import-id

5.11

stack-data

0.2.0

statsmodels

0.14.0

tenacity

8.2.2

threadpoolctl

2.2.0

tokenize-rt

4.2.1

tornado

6.3.2

traitlets

5.7.1

typing_extensions

4.10.0

tzdata

2022.1

ujson

5.4.0

unattended-upgrades

0.1

urllib3

1.26.16

virtualenv

20.24.2

wadllib

1.3.6

wcwidth

0.2.5

wheel

0.38.4

zipp

3.11.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot.

Library

Version

Library

Version

Library

Version

arrow

14.0.0.2

askpass

1.2.0

assertthat

0.2.1

backports

1.4.1

base

4.3.2

base64enc

0.1-3

bigD

0.2.0

bit

4.0.5

bit64

4.0.5

bitops

1.0-7

blob

1.2.4

boot

1.3-28

brew

1.0-10

brio

1.1.4

broom

1.0.5

bslib

0.6.1

cachem

1.0.8

callr

3.7.3

caret

6.0-94

cellranger

1.1.0

chron

2.3-61

class

7.3-22

cli

3.6.2

clipr

0.8.0

clock

0.7.0

cluster

2.1.4

codetools

0.2-19

colorspace

2.1-0

commonmark

1.9.1

compiler

4.3.2

config

0.3.2

conflicted

1.2.0

cpp11

0.4.7

crayon

1.5.2

credentials

2.0.1

curl

5.2.0

data.table

1.15.0

datasets

4.3.2

DBI

1.2.1

dbplyr

2.4.0

desc

1.4.3

devtools

2.4.5

diagram

1.6.5

diffobj

0.3.5

digest

0.6.34

downlit

0.4.3

dplyr

1.1.4

dtplyr

1.3.1

e1071

1.7-14

ellipsis

0.3.2

evaluate

0.23

fansi

1.0.6

farver

2.1.1

fastmap

1.1.1

fontawesome

0.5.2

forcats

1.0.0

foreach

1.5.2

foreign

0.8-85

forge

0.2.0

fs

1.6.3

future

1.33.1

future.apply

1.11.1

gargle

1.5.2

generics

0.1.3

gert

2.0.1

ggplot2

3.4.4

gh

1.4.0

git2r

0.33.0

gitcreds

0.1.2

glmnet

4.1-8

globals

0.16.2

glue

1.7.0

googledrive

2.1.1

googlesheets4

1.1.1

gower

1.0.1

graphics

4.3.2

grDevices

4.3.2

grid

4.3.2

gridExtra

2.3

gsubfn

0.7

gt

0.10.1

gtable

0.3.4

hardhat

1.3.1

haven

2.5.4

highr

0.10

hms

1.1.3

htmltools

0.5.7

htmlwidgets

1.6.4

httpuv

1.6.14

httr

1.4.7

httr2

1.0.0

ids

1.0.1

ini

0.3.1

ipred

0.9-14

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

1.8.8

juicyjuice

0.1.0

KernSmooth

2.23-21

knitr

1.45

labeling

0.4.3

later

1.3.2

lattice

0.21-8

lava

1.7.3

lifecycle

1.0.4

listenv

0.9.1

lubridate

1.9.3

magrittr

2.0.3

markdown

1.12

MASS

7.3-60

Matrix

1.5-4.1

memoise

2.0.1

methods

4.3.2

mgcv

1.8-42

mime

0.12

miniUI

0.1.1.1

mlflow

2.10.0

ModelMetrics

1.2.2.2

modelr

0.1.11

munsell

0.5.0

nlme

3.1-163

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.1.1

parallel

4.3.2

parallelly

1.36.0

pillar

1.9.0

pkgbuild

1.4.3

pkgconfig

2.0.3

pkgdown

2.0.7

pkgload

1.3.4

plogr

0.2.0

plyr

1.8.9

praise

1.0.0

prettyunits

1.2.0

pROC

1.18.5

processx

3.8.3

prodlim

2023.08.28

profvis

0.3.8

progress

1.2.3

progressr

0.14.0

promises

1.2.1

proto

1.0.0

proxy

0.4-27

ps

1.7.6

purrr

1.0.2

R6

2.5.1

ragg

1.2.7

randomForest

4.7-1.1

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.0.12

RcppEigen

0.3.3.9.4

reactable

0.4.4

reactR

0.5.0

readr

2.1.5

readxl

1.4.3

recipes

1.0.9

rematch

2.0.0

rematch2

2.1.2

remotes

2.4.2.1

reprex

2.1.0

reshape2

1.4.4

rlang

1.1.3

rmarkdown

2.25

RODBC

1.3-23

roxygen2

7.3.1

rpart

4.1.21

rprojroot

2.0.4

Rserve

1.8-13

RSQLite

2.3.5

rstudioapi

0.15.0

rversions

2.1.2

rvest

1.0.3

sass

0.4.8

scales

1.3.0

selectr

0.4-2

sessioninfo

1.2.2

shape

1.4.6

shiny

1.8.0

sourcetools

0.1.7-1

sparklyr

1.8.4

spatial

7.3-15

splines

4.3.2

sqldf

0.4-11

SQUAREM

2021.1

stats

4.3.2

stats4

4.3.2

stringi

1.8.3

stringr

1.5.1

survival

3.5-5

swagger

3.33.1

sys

3.4.2

systemfonts

1.0.5

tcltk

4.3.2

testthat

3.2.1

textshaping

0.3.7

tibble

3.2.1

tidyr

1.3.1

tidyselect

1.2.0

tidyverse

2.0.0

timechange

0.3.0

timeDate

4032.109

tinytex

0.49

tools

4.3.2

tzdb

0.4.0

urlchecker

1.0.1

usethis

2.2.2

utf8

1.2.4

utils

4.3.2

uuid

1.2-0

V8

4.4.1

vctrs

0.6.5

viridisLite

0.4.2

vroom

1.6.5

waldo

0.5.2

whisker

0.4.1

withr

3.0.0

xfun

0.41

xml2

1.3.6

xopen

1.0.0

xtable

1.8-4

yaml

2.3.8

zeallot

0.1.0

zip

2.3.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.12.0

com.amazonaws

aws-java-sdk-autoscaling

1.12.610

com.amazonaws

aws-java-sdk-cloudformation

1.12.610

com.amazonaws

aws-java-sdk-cloudfront

1.12.610

com.amazonaws

aws-java-sdk-cloudhsm

1.12.610

com.amazonaws

aws-java-sdk-cloudsearch

1.12.610

com.amazonaws

aws-java-sdk-cloudtrail

1.12.610

com.amazonaws

aws-java-sdk-cloudwatch

1.12.610

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.610

com.amazonaws

aws-java-sdk-codedeploy

1.12.610

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.610

com.amazonaws

aws-java-sdk-cognitosync

1.12.610

com.amazonaws

aws-java-sdk-config

1.12.610

com.amazonaws

aws-java-sdk-core

1.12.610

com.amazonaws

aws-java-sdk-datapipeline

1.12.610

com.amazonaws

aws-java-sdk-directconnect

1.12.610

com.amazonaws

aws-java-sdk-directory

1.12.610

com.amazonaws

aws-java-sdk-dynamodb

1.12.610

com.amazonaws

aws-java-sdk-ec2

1.12.610

com.amazonaws

aws-java-sdk-ecs

1.12.610

com.amazonaws

aws-java-sdk-efs

1.12.610

com.amazonaws

aws-java-sdk-elasticache

1.12.610

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.610

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.610

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.610

com.amazonaws

aws-java-sdk-emr

1.12.610

com.amazonaws

aws-java-sdk-glacier

1.12.610

com.amazonaws

aws-java-sdk-glue

1.12.610

com.amazonaws

aws-java-sdk-iam

1.12.610

com.amazonaws

aws-java-sdk-importexport

1.12.610

com.amazonaws

aws-java-sdk-kinesis

1.12.610

com.amazonaws

aws-java-sdk-kms

1.12.610

com.amazonaws

aws-java-sdk-lambda

1.12.610

com.amazonaws

aws-java-sdk-logs

1.12.610

com.amazonaws

aws-java-sdk-machinelearning

1.12.610

com.amazonaws

aws-java-sdk-opsworks

1.12.610

com.amazonaws

aws-java-sdk-rds

1.12.610

com.amazonaws

aws-java-sdk-redshift

1.12.610

com.amazonaws

aws-java-sdk-route53

1.12.610

com.amazonaws

aws-java-sdk-s3

1.12.610

com.amazonaws

aws-java-sdk-ses

1.12.610

com.amazonaws

aws-java-sdk-simpledb

1.12.610

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.610

com.amazonaws

aws-java-sdk-sns

1.12.610

com.amazonaws

aws-java-sdk-sqs

1.12.610

com.amazonaws

aws-java-sdk-ssm

1.12.610

com.amazonaws

aws-java-sdk-storagegateway

1.12.610

com.amazonaws

aws-java-sdk-sts

1.12.610

com.amazonaws

aws-java-sdk-support

1.12.610

com.amazonaws

aws-java-sdk-swf-libraries

1.11.22

com.amazonaws

aws-java-sdk-workspaces

1.12.610

com.amazonaws

jmespath-java

1.12.610

com.clearspring.analytics

stream

2.9.6

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.27.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

compilerplugin_2.12

0.4.15-10

com.databricks.scalapb

scalapb-runtime_2.12

0.4.15-10

com.esotericsoftware

kryo-shaded

4.0.2

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.3.4

com.fasterxml.jackson.core

jackson-annotations

2.15.2

com.fasterxml.jackson.core

jackson-core

2.15.2

com.fasterxml.jackson.core

jackson-databind

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.15.2

com.fasterxml.jackson.dataformat

jackson-dataformat-yaml

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.16.0

com.fasterxml.jackson.module

jackson-module-paranamer

2.15.2

com.fasterxml.jackson.module

jackson-module-scala_2.12

2.15.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.5-4

com.github.wendykierp

JTransforms

3.1

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.10.1

com.google.crypto.tink

tink

1.9.0

com.google.errorprone

error_prone_annotations

2.10.0

com.google.flatbuffers

flatbuffers-java

23.5.26

com.google.guava

guava

15.0

com.google.protobuf

protobuf-java

2.6.1

com.helger

profiler

1.1.1

com.ibm.icu

icu4j

75.1

com.jcraft

jsch

0.1.55

com.jolbox

bonecp

0.8.0.RELEASE

com.lihaoyi

sourcecode_2.12

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.9

com.microsoft.sqlserver

mssql-jdbc

11.2.2.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.12

0.4.12

com.twitter

chill-java

0.10.0

com.twitter

chill_2.12

0.10.0

com.twitter

util-app_2.12

7.1.0

com.twitter

util-core_2.12

7.1.0

com.twitter

util-function_2.12

7.1.0

com.twitter

util-jvm_2.12

7.1.0

com.twitter

util-lint_2.12

7.1.0

com.twitter

util-registry_2.12

7.1.0

com.twitter

util-stats_2.12

7.1.0

com.typesafe

config

1.4.3

com.typesafe.scala-logging

scala-logging_2.12

3.7.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

commons-cli

commons-cli

1.5.0

commons-codec

commons-codec

1.16.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.5

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.13.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.3

dev.ludovic.netlib

blas

3.0.3

dev.ludovic.netlib

lapack

3.0.3

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

0.25

io.delta

delta-sharing-client_2.12

1.1.1

io.dropwizard.metrics

metrics-annotation

4.2.19

io.dropwizard.metrics

metrics-core

4.2.19

io.dropwizard.metrics

metrics-graphite

4.2.19

io.dropwizard.metrics

metrics-healthchecks

4.2.19

io.dropwizard.metrics

metrics-jetty9

4.2.19

io.dropwizard.metrics

metrics-jmx

4.2.19

io.dropwizard.metrics

metrics-json

4.2.19

io.dropwizard.metrics

metrics-jvm

4.2.19

io.dropwizard.metrics

metrics-servlets

4.2.19

io.netty

netty-all

4.1.96.Final

io.netty

netty-buffer

4.1.96.Final

io.netty

netty-codec

4.1.96.Final

io.netty

netty-codec-http

4.1.96.Final

io.netty

netty-codec-http2

4.1.96.Final

io.netty

netty-codec-socks

4.1.96.Final

io.netty

netty-common

4.1.96.Final

io.netty

netty-handler

4.1.96.Final

io.netty

netty-handler-proxy

4.1.96.Final

io.netty

netty-resolver

4.1.96.Final

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.61.Final-windows-x86_64

io.netty

netty-tcnative-classes

2.0.61.Final

io.netty

netty-transport

4.1.96.Final

io.netty

netty-transport-classes-epoll

4.1.96.Final

io.netty

netty-transport-classes-kqueue

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.1.96.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.1.96.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.1.96.Final

io.prometheus

simpleclient

0.7.0

io.prometheus

simpleclient_common

0.7.0

io.prometheus

simpleclient_dropwizard

0.7.0

io.prometheus

simpleclient_pushgateway

0.7.0

io.prometheus

simpleclient_servlet

0.7.0

io.prometheus.jmx

collector

0.12.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.12.1

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.3

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.9.3

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.10.11

org.apache.ant

ant-jsch

1.10.11

org.apache.ant

ant-launcher

1.10.11

org.apache.arrow

arrow-format

15.0.0

org.apache.arrow

arrow-memory-core

15.0.0

org.apache.arrow

arrow-memory-netty

15.0.0

org.apache.arrow

arrow-vector

15.0.0

org.apache.avro

avro

1.11.3

org.apache.avro

avro-ipc

1.11.3

org.apache.avro

avro-mapred

1.11.3

org.apache.commons

commons-collections4

4.4

org.apache.commons

commons-compress

1.23.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.12.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.10.0

org.apache.curator

curator-client

2.13.0

org.apache.curator

curator-framework

2.13.0

org.apache.curator

curator-recipes

2.13.0

org.apache.datasketches

datasketches-java

3.1.0

org.apache.datasketches

datasketches-memory

2.0.0

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.3.6

org.apache.hive

hive-beeline

2.3.9

org.apache.hive

hive-cli

2.3.9

org.apache.hive

hive-jdbc

2.3.9

org.apache.hive

hive-llap-client

2.3.9

org.apache.hive

hive-llap-common

2.3.9

org.apache.hive

hive-serde

2.3.9

org.apache.hive

hive-shims

2.3.9

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.9

org.apache.hive.shims

hive-shims-common

2.3.9

org.apache.hive.shims

hive-shims-scheduler

2.3.9

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.1

org.apache.logging.log4j

log4j-1.2-api

2.22.1

org.apache.logging.log4j

log4j-api

2.22.1

org.apache.logging.log4j

log4j-core

2.22.1

org.apache.logging.log4j

log4j-layout-template-json

2.22.1

org.apache.logging.log4j

log4j-slf4j2-impl

2.22.1

org.apache.orc

orc-core

1.9.2-shaded-protobuf

org.apache.orc

orc-mapreduce

1.9.2-shaded-protobuf

org.apache.orc

orc-shims

1.9.2

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.12.0

org.apache.ws.xmlschema

xmlschema-core

2.3.0

org.apache.xbean

xbean-asm9-shaded

4.23

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.6.3

org.apache.zookeeper

zookeeper-jute

3.6.3

org.checkerframework

checker-qual

3.31.0

org.codehaus.jackson

jackson-core-asl

1.9.13

org.codehaus.jackson

jackson-mapper-asl

1.9.13

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.collections

eclipse-collections

11.1.0

org.eclipse.collections

eclipse-collections-api

11.1.0

org.eclipse.jetty

jetty-client

9.4.52.v20230823

org.eclipse.jetty

jetty-continuation

9.4.52.v20230823

org.eclipse.jetty

jetty-http

9.4.52.v20230823

org.eclipse.jetty

jetty-io

9.4.52.v20230823

org.eclipse.jetty

jetty-jndi

9.4.52.v20230823

org.eclipse.jetty

jetty-plus

9.4.52.v20230823

org.eclipse.jetty

jetty-proxy

9.4.52.v20230823

org.eclipse.jetty

jetty-security

9.4.52.v20230823

org.eclipse.jetty

jetty-server

9.4.52.v20230823

org.eclipse.jetty

jetty-servlet

9.4.52.v20230823

org.eclipse.jetty

jetty-servlets

9.4.52.v20230823

org.eclipse.jetty

jetty-util

9.4.52.v20230823

org.eclipse.jetty

jetty-util-ajax

9.4.52.v20230823

org.eclipse.jetty

jetty-webapp

9.4.52.v20230823

org.eclipse.jetty

jetty-xml

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-api

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-client

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-common

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-server

9.4.52.v20230823

org.eclipse.jetty.websocket

websocket-servlet

9.4.52.v20230823

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.40

org.glassfish.jersey.containers

jersey-container-servlet-core

2.40

org.glassfish.jersey.core

jersey-client

2.40

org.glassfish.jersey.core

jersey-common

2.40

org.glassfish.jersey.core

jersey-server

2.40

org.glassfish.jersey.inject

jersey-hk2

2.40

org.hibernate.validator

hibernate-validator

6.1.7.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.3.2.Final

org.jdbi

jdbi

2.63.1

org.jetbrains

annotations

17.0.0

org.joda

joda-convert

1.7

org.jodd

jodd-core

3.5.2

org.json4s

json4s-ast_2.12

3.7.0-M11

org.json4s

json4s-core_2.12

3.7.0-M11

org.json4s

json4s-jackson_2.12

3.7.0-M11

org.json4s

json4s-scalap_2.12

3.7.0-M11

org.lz4

lz4-java

1.8.0

org.mlflow

mlflow-spark_2.12

2.9.1

org.objenesis

objenesis

2.5.1

org.postgresql

postgresql

42.6.1

org.roaringbitmap

RoaringBitmap

0.9.45-databricks

org.roaringbitmap

shims

0.9.45-databricks

org.rocksdb

rocksdbjni

8.11.4

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.12

2.12.15

org.scala-lang

scala-library_2.12

2.12.15

org.scala-lang

scala-reflect_2.12

2.12.15

org.scala-lang.modules

scala-collection-compat_2.12

2.11.0

org.scala-lang.modules

scala-parser-combinators_2.12

1.1.2

org.scala-lang.modules

scala-xml_2.12

1.2.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.12

1.14.2

org.scalactic

scalactic_2.12

3.2.15

org.scalanlp

breeze-macros_2.12

2.1.0

org.scalanlp

breeze_2.12

2.1.0

org.scalatest

scalatest-compatible

3.2.15

org.scalatest

scalatest-core_2.12

3.2.15

org.scalatest

scalatest-diagrams_2.12

3.2.15

org.scalatest

scalatest-featurespec_2.12

3.2.15

org.scalatest

scalatest-flatspec_2.12

3.2.15

org.scalatest

scalatest-freespec_2.12

3.2.15

org.scalatest

scalatest-funspec_2.12

3.2.15

org.scalatest

scalatest-funsuite_2.12

3.2.15

org.scalatest

scalatest-matchers-core_2.12

3.2.15

org.scalatest

scalatest-mustmatchers_2.12

3.2.15

org.scalatest

scalatest-propspec_2.12

3.2.15

org.scalatest

scalatest-refspec_2.12

3.2.15

org.scalatest

scalatest-shouldmatchers_2.12

3.2.15

org.scalatest

scalatest-wordspec_2.12

3.2.15

org.scalatest

scalatest_2.12

3.2.15

org.slf4j

jcl-over-slf4j

2.0.7

org.slf4j

jul-to-slf4j

2.0.7

org.slf4j

slf4j-api

2.0.7

org.slf4j

slf4j-simple

1.7.25

org.threeten

threeten-extra

1.7.1

org.tukaani

xz

1.9

org.typelevel

algebra_2.12

2.0.1

org.typelevel

cats-kernel_2.12

2.1.1

org.typelevel

spire-macros_2.12

0.17.0

org.typelevel

spire-platform_2.12

0.17.0

org.typelevel

spire-util_2.12

0.17.0

org.typelevel

spire_2.12

0.17.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

1.6.2-linux-x86_64

software.amazon.ion

ion-java

1.0.2

stax

stax-api

1.0.1