Databricks Runtime 11.0 (EoS)
Note
Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.
The following release notes provide information about Databricks Runtime 11.0, powered by Apache Spark 3.3.0. Databricks released this version in June 2022.
New features and improvements
Synapse connector now writes Parquet data in non-legacy mode
Dropping a missing Delta table constraint now raises an error
SQL: New
EXCEPT
clause inSELECT
statement excludes columns from selectionSupport for dropping columns in Delta tables (Public Preview)
CONVERT TO DELTA
is now supported in Unity Catalog enabled environments (Public Preview)
New version of Apache Spark
Databricks Runtime 11.0 and Databricks Runtime 11.0 Photon include Apache Spark 3.3.0. For details, see Apache Spark.
Python notebooks now use the IPython kernel
In Databricks Runtime 11.0 and above, Python notebooks use the IPython kernel to execute Python code. See IPython kernel.
Support for ipywidgets
You can now use ipywidgets to make your Databricks Python notebooks interactive. See ipywidgets.
Synapse connector now writes Parquet data in non-legacy mode
The Azure Synapse connector now writes Parquet data in non-legacy mode. It preserves the INT96
timestamp format when using PolyBase and COPY
commands for both batch and streaming workloads.
HTTPS scheme now enforced when ABFS client uses SAS token
When the Azure Blob File System (ABFS) client uses a Shared Access Signature (SAS) token, the HTTPS scheme is now enforced.
SQL: DESC
is now an alias for DESCRIBE
You can now use DESC
as an alias for DESCRIBE
when you describe external locations or storage credentials. For example:
-- Describe an external location.
DESC EXTERNAL LOCATION location_name;
-- Describe a storage credential.
DESC STORAGE CREDENTIAL credential_name;
SQL: New current_version
function outputs version details
The new current_version
function outputs the current Databricks Runtime version if available, the current Databricks SQL version if available, and other related version details. Use this new function to query version-related information. See current_version function.
Dropping a missing Delta table constraint now raises an error
If you now try to drop a Delta table constraint by name, and that constraint does not exist, you will get an error. To get the previous behavior, which does not throw an error if the constraint does not exist, you must now use the IF EXISTS
statement. See ALTER TABLE.
SQL: New EXCEPT
clause in SELECT
statement excludes columns from selection
SELECT
statements now support the EXCEPT
clause to exclude columns from selection. For example, SELECT * EXCEPT (x) FROM table
returns all of table
’s columns except x
. Nested columns are also allowed. For example, SELECT * EXCEPT (x.a) FROM table
returns all of table
’s columns but omits field a
from struct x
.
Support for dropping columns in Delta tables (Public Preview)
You can use ALTER TABLE <table-name> DROP COLUMN [IF EXISTS] <column-name>
or ALTER TABLE <table-name> DROP COLUMNS [IF EXISTS] (<column-name>, *)
to drop a column or a list of columns, respectively, from a Delta table as a metadata-only operation. The columns are effectively “soft-deleted,” as they are still in the underlying Parquet files but are no longer visible to the Delta table.
You can use REORG TABLE <table-name> APPLY (PURGE)
to trigger a file rewrite on the files that contain any soft-deleted data such as dropped columns.
You can use VACUUM
to remove the deleted files from physical storage, including the old files that contain dropped columns and have been rewritten by REORG TABLE
.
COPY INTO
improvements
You can now create empty placeholder Delta tables so that the schema is later inferred during a COPY INTO
command:
CREATE TABLE IF NOT EXISTS my_table
[COMMENT <table-description>]
[TBLPROPERTIES (<table-properties>)];
COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('mergeSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');
The preceding SQL statement is idempotent and can be scheduled to run to ingest data exactly once into a Delta table.
Note
The empty Delta table is not usable outside of COPY INTO
. You cannot use INSERT INTO
and MERGE INTO
to write data into schema-less Delta tables. After data is inserted into the table with COPY INTO
, the table is queryable.
If the data you are ingesting cannot be read due to some corruption issue, you can skip corrupted files by setting ignoreCorruptFiles
to true
in the FORMAT_OPTIONS
:
COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('ignoreCorruptFiles' = 'true')
The COPY INTO
command returns the number of files that were skipped due to corruption in the num_skipped_corrupt_files
column. This metric also appears in the operationMetrics
column under numSkippedCorruptFiles
after you run DESCRIBE HISTORY
on the Delta table.
Corrupt files are not tracked by COPY INTO
, so they can be reloaded in a subsequent run if the corruption is fixed. You can see which files are corrupt by running COPY INTO
in VALIDATE
mode.
CONVERT TO DELTA
is now supported in Unity Catalog enabled environments (Public Preview)
In Unity-Catalog-enabled environments, with CONVERT TO DELTA
you can now:
Convert Parquet files in external locations to Delta Lake.
Convert Parquet external tables to Delta tables.
Behavior changes
SQL: lpad
and rpad
functions now support byte sequences
The lpad and rpad functions have been updated to add support for byte sequences in addition to strings.
String format in format_string
and printf
no longer allows %0$
Specifying the format %0$
in the format_string
and printf
functions now throws an error by default. This change is to preserve expected behavior with previous versions of the Databricks Runtime and common third-party databases. The first argument should always reference %1$
when you use an argument index to indicate the position of the argument in the argument list.
Null values in CSV files are now written as unquoted empty strings by default
Null values in CSV files were previously written as quoted empty strings. With this release, null values in CSV files are now written as unquoted empty strings by default. To change back to the previous behavior, set the nullValue
option to ""
for write operations.
The table property external
is now reserved
The property external
is now a reserved table property by default. Exceptions are now thrown when you use the external
property with the CREATE TABLE ... TBLPROPERTIES
and ALTER TABLE ... SET TBLPROPERTIES
clauses.
Log4j is upgraded from Log4j 1 to Log4j 2
Log4j 1 is being upgraded to Log4j 2. Legacy Log4j 1 dependencies are being removed.
If you depend on the Log4j 1 classes that were previously included inside the Databricks Runtime, those classes no longer exist. You should upgrade your dependencies to Log4j 2.
If you have custom plugins or configuration files that depend on Log4j 2, they may no longer work with the version of Log4j 2 in this release. For assistance, contact your Databricks account team.
Libraries installed from Maven are now resolved in your compute plane by default
Maven libraries now resolve in your compute plane by default when you install libraries on a cluster. You cluster must have access to Maven Central. Alternatively, you can revert to the previous behavior by setting the Spark configuration property:
spark.databricks.libraries.enableMavenResolution false
Bug fixes
Binary compatibility for
LeafNode
,UnaryNode
, andBinaryNode
between Apache Spark and the Databricks Runtime has been fixed and the classes are now compatible with Apache Spark 3.3.0 and above. If you encounter the following or a similar message when using a third-party package with the Databricks Runtime, rebuild the package with Apache Spark 3.3.0 or above:Found interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode, but class was expected
.
Library upgrades
Upgraded Python libraries:
platformdirs from 2.5.1 to 2.5.2
protobuf from 3.20.0 to 3.20.1
Upgraded R libraries:
blob from 1.2.2 to 1.2.3
broom from 0.7.12 to 0.8.0
caret from 6.0-91 to 6.0-92
cli from 3.2.0 to 3.3.0
dplyr from 1.0.8 to 1.0.9
future from 1.24.0 to 1.25.0
future.apply from 1.8.1 to 1.9.0
gert from 1.5.0 to 1.6.0
ggplot2 from 3.3.5 to 3.3.6
glmnet from 4.1-3 to 4.1-4
haven from 2.4.3 to 2.5.0
httr from 1.4.2 to 1.4.3
knitr from 1.38 to 1.39
magrittr from 2.0.2 to 2.0.3
parallelly from 1.30.0 to 1.31.1
ps from 1.6.0 to 1.7.0
RColorBrewer from 1.1-2 to 1.1-3
RcppEigen from 0.3.3.9.1 to 0.3.3.9.2
readxl from 1.3.1 to 1.4.0
rmarkdown from 2.13 to 2.14
rprojroot from 2.0.2 to 2.0.3
RSQLite from 2.2.11 to 2.2.13
scales from 1.1.1 to 1.2.0
testthat from 3.1.2 to 3.1.4
tibble from 3.1.6 to 3.1.7
tinytex from 0.37 to 0.38
tzdb from 0.2.0 to 0.3.0
uuid from 1.0-4 to 1.1-0
vctrs from 0.3.8 to 0.4.1
Upgraded Java libraries:
com.fasterxml.jackson.core.jackson-annotations from 2.13.0 to 2.13.3
com.fasterxml.jackson.core.jackson-core from 2.13.0 to 2.13.3
com.fasterxml.jackson.core.jackson-databind from 2.13.0 to 2.13.3
com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.0 to 2.13.3
com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.0 to 2.13.3
com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.0 to 2.13.3
com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.0 to 2.13.3
com.google.crypto.tink.tink from 1.6.0 to 1.6.1
com.ning.compress-lzf from 1.0.3 to 1.1
dev.ludovic.netlib.arpack from 2.2.0 to 2.2.1
dev.ludovic.netlib.blas from 2.2.0 to 2.2.1
dev.ludovic.netlib.lapack from 2.2.0 to 2.2.1
io.netty.netty-all from 4.1.73.Final to 4.1.74.Final
io.netty.netty-buffer from 4.1.73.Final to 4.1.74.Final
io.netty.netty-codec from 4.1.73.Final to 4.1.74.Final
io.netty.netty-common from 4.1.73.Final to 4.1.74.Final
io.netty.netty-handler from 4.1.73.Final to 4.1.74.Final
io.netty.netty-resolver from 4.1.73.Final to 4.1.74.Final
io.netty.netty-tcnative-classes from 2.0.46.Final to 2.0.48.Final
io.netty.netty-transport from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-classes-epoll from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-classes-kqueue from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-native-epoll-linux-aarch_64 from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-native-epoll-linux-x86_64 from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-native-kqueue-osx-aarch_64 from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-native-kqueue-osx-x86_64 from 4.1.73.Final to 4.1.74.Final
io.netty.netty-transport-native-unix-common from 4.1.73.Final to 4.1.74.Final
joda-time.joda-time from 2.10.12 to 2.10.13
org.apache.commons.commons-math3 from 3.4.1 to 3.6.1
org.apache.httpcomponents.httpcore from 4.4.12 to 4.4.14
org.apache.orc.orc-core from 1.7.3 to 1.7.4
org.apache.orc.orc-mapreduce from 1.7.3 to 1.7.4
org.apache.orc.orc-shims from 1.7.3 to 1.7.4
org.eclipse.jetty.jetty-client from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-continuation from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-http from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-io from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-jndi from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-plus from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-proxy from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-security from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-server from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-servlet from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-servlets from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-util from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-util-ajax from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-webapp from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.jetty-xml from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.websocket.websocket-api from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.websocket.websocket-client from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.websocket.websocket-common from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.websocket.websocket-server from 9.4.43.v20210629 to 9.4.46.v20220331
org.eclipse.jetty.websocket.websocket-servlet from 9.4.43.v20210629 to 9.4.46.v20220331
org.mariadb.jdbc.mariadb-java-client from 2.2.5 to 2.7.4
org.postgresql.postgresql from 42.2.19 to 42.3.3
org.roaringbitmap.RoaringBitmap from 0.9.23 to 0.9.25
org.roaringbitmap.shims from 0.9.23 to 0.9.25
org.rocksdb.rocksdbjni from 6.20.3 to 6.24.2
org.slf4j.jcl-over-slf4j from 1.7.32 to 1.7.36
org.slf4j.jul-to-slf4j from 1.7.32 to 1.7.36
org.slf4j.slf4j-api from 1.7.30 to 1.7.36
Apache Spark
Databricks Runtime 11.0 includes Apache Spark 3.3.0.
In this section:
Spark SQL and Core
ANSI mode
New explicit cast syntax rules in ANSI mode (SPARK-33354)
Elt() should return null if index is null under ANSI mode (SPARK-38304)
Optionally return null result if element not exists in array/map (SPARK-37750)
Allow casting between numeric type and timestamp type (SPARK-37714)
Disable ANSI reserved keywords by default (SPARK-37724)
Use store assignment rules for resolving function invocation (SPARK-37438)
Add a config to allow casting between Datetime and Numeric (SPARK-37179)
Add a config to optionally enforce ANSI reserved keywords (SPARK-37133)
Disallow binary operations between Interval and String literal (SPARK-36508)
Feature enhancements
Support ANSI SQL INTERVAL types (SPARK-27790)
Error Message Improvements (SPARK-38781)
Hidden File Metadata Support for Spark SQL (SPARK-37273)
Support raw string literal (SPARK-36371)
Helper class for batch Dataset.observe() (SPARK-34806)
Support specify initial partition number for rebalance (SPARK-38410)
Support cascade mode for
dropNamespace
API (SPARK-37929)Allow store assignment and implicit cast among datetime types (SPARK-37707)
Collect, first and last should be deterministic aggregate functions (SPARK-32940)
Add ExpressionBuilder for functions with complex overloads (SPARK-37164)
Add array support to union by name (SPARK-36546)
Add df.withMetadata: a syntax sugar to update the metadata of a dataframe (SPARK-36642)
Support raw string literal (SPARK-36371)
Use CAST in parsing of dates/timestamps with default pattern (SPARK-36418)
Support value class in nested schema for Dataset (SPARK-20384)
Add AS OF syntax support (SPARK-37219)
Add REPEATABLE in TABLESAMPLE to specify seed (SPARK-37165)
Add ansi syntax
set catalog xxx
to change the current catalog (SPARK-36841)Support ILIKE (ALL | ANY | SOME) - case insensitive LIKE (SPARK-36674, SPARK-36736, SPARK-36778)
Support query stage show runtime statistics in formatted explain mode (SPARK-38322)
Add spill size metrics for sort merge join (SPARK-37726)
Update the SQL syntax of SHOW FUNCTIONS (SPARK-37777)
Support DROP COLUMN [IF EXISTS] syntax (SPARK-38939)
New built-in functions and their extensions (SPARK-38783)
Datetime
Add the TIMESTAMPADD() function (SPARK-38195)
Add the TIMESTAMPDIFF() function (SPARK-38284)
Add the
DATEDIFF()
alias forTIMESTAMPDIFF()
(SPARK-38389)Add the
DATEADD()
alias forTIMESTAMPADD()
(SPARK-38332)Add the
convert\_timezone()
function (SPARK-37552, SPARK-37568)Expose make_date expression in functions.scala (SPARK-36554)
AES functions (SPARK-12567)
Add aes_encrypt and aes_decrypt builtin functions (SPARK-12567)
Support the GCM mode by
aes\_encrypt()
/aes\_decrypt()
(SPARK-37591)Set
GCM
as the default mode inaes\_encrypt()
/aes\_decrypt()
(SPARK-37666)Add the
mode
andpadding
args toaes\_encrypt()
/aes\_decrypt()
(SPARK-37586)
ANSI Aggregation Function (SPARK-37671)
Support ANSI Aggregate Function: regr_count (SPARK-37613)
Support ANSI Aggregate Function: regr_avgx & regr_avgy (SPARK-37614)
Support ANSI Aggregation Function: percentile_cont (SPARK-37676, SPARK-38219)
Support ANSI Aggregation Function: percentile_disc (SPARK-37691)
Support ANSI Aggregate Function: regr_count (SPARK-37613)
Support ANSI Aggregate Function: array_agg (SPARK-27974)
Support ANSI Aggregate Function: regr_r2 (SPARK-37641)
New SQL function: try_avg (SPARK-38589)
Collections
Introduce SQL function ARRAY_SIZE (SPARK-38345)
New SQL function: map_contains_key (SPARK-37584)
New SQL function: try_element_at (SPARK-37533)
New SQL function: try_sum (SPARK-38548)
Format
Add a new SQL function to_binary (SPARK-37507, SPARK-38796)
New SQL function: try_to_binary (SPARK-38590, SPARK-38796)
Data Type Formatting Functions:
to\_number
(SPARK-28137)
String/Binary
Add CONTAINS() string function (SPARK-37508)
Add the
startswith()
andendswith()
string functions (SPARK-37520)Add lpad and rpad functions for binary strings (SPARK-37047)
Support split_part Function (SPARK-38063)
Add scale parameter to floor and ceil functions (SPARK-37475)
New SQL functions: try_subtract and try_multiply (SPARK-38164)
Implements histogram_numeric aggregation function which supports partial aggregation (SPARK-16280)
Add max_by/min_by to sql.functions (SPARK-36963)
Add new built-in SQL functions: SEC and CSC (SPARK-36683)
array_intersect handles duplicated Double.NaN and Float.NaN (SPARK-36754)
Add cot as Scala and Python functions (SPARK-36660)
Performance enhancements
Whole-stage code generation
Add code-gen for sort aggregate without grouping keys (SPARK-37564)
Add code-gen for full outer sort merge join (SPARK-35352)
Add code-gen for full outer shuffled hash join (SPARK-32567)
Add code-gen for existence sort merge join (SPARK-37316)
Push down (filters)
Push down filters through RebalancePartitions (SPARK-37828)
Push down boolean column filter (SPARK-36644)
Push down limit 1 for right side of left semi/anti join if join condition is empty (SPARK-37917)
Translate more standard aggregate functions for pushdown (SPARK-37527)
Support propagate empty relation through aggregate/union (SPARK-35442)
Row-level Runtime Filtering (SPARK-32268)
Support Left Semi join in row level runtime filters (SPARK-38565)
Support predicate pushdown and column pruning for de-duped CTEs (SPARK-37670)
Vectorization
Implement a ConstantColumnVector and improve performance of the hidden file metadata (SPARK-37896)
Enable vectorized read for VectorizedPlainValuesReader.readBooleans (SPARK-35867)
Combine/remove/replace nodes
Combine unions if there is a project between them (SPARK-37915)
Combine to one cast if we can safely up-cast two casts (SPARK-37922)
Remove the Sort if it is the child of RepartitionByExpression (SPARK-36703)
Removes outer join if it only has DISTINCT on streamed side with alias (SPARK-37292)
Replace hash with sort aggregate if child is already sorted (SPARK-37455)
Only collapse projects if we don’t duplicate expensive expressions (SPARK-36718)
Remove redundant aliases after RewritePredicateSubquery (SPARK-36280)
Merge non-correlated scalar subqueries (SPARK-34079)
Partitioning
Do not add dynamic partition pruning if there exists static partition pruning (SPARK-38148)
Improve RebalancePartitions in rules of Optimizer (SPARK-37904)
Add small partition factor for rebalance partitions (SPARK-37357)
Join
Fine tune logic to demote Broadcast hash join in DynamicJoinSelection (SPARK-37753)
Ignore duplicated join keys when building relation for SEMI/ANTI shuffled hash join (SPARK-36794)
Support optimize skewed join even if introduce extra shuffle (SPARK-33832)
AQE
Support eliminate limits in AQE Optimizer (SPARK-36424)
Optimize one row plan in normal and AQE Optimizer (SPARK-38162)
Aggregate.groupOnly support foldable expressions (SPARK-38489)
ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform (SPARK-37796)
Add tree pattern pruning to CTESubstitution rule (SPARK-37379)
Add more Not operator simplifications (SPARK-36665)
Support BooleanType in UnwrapCastInBinaryComparison (SPARK-36607)
Coalesce drop all expressions after the first non nullable expression (SPARK-36359)
Add a logical plan visitor to propagate the distinct attributes (SPARK-36194)
Built-in connector enhancements
General
Lenient serialization of datetime from datasource (SPARK-38437)
Treat table location as absolute when the first letter of its path is slash in create/alter table (SPARK-38236)
Remove leading zeros from empty static number type partition (SPARK-35561)
Support
ignoreCorruptFiles
andignoreMissingFiles
in Data Source options (SPARK-38767)Add command
SHOW CATALOGS
(SPARK-35973)
Parquet
Enable matching schema column names by field ids (SPARK-38094)
Remove check field name when reading/writing data in parquet (SPARK-27442)
Support vectorized read boolean values use RLE encoding with Parquet DataPage V2 (SPARK-37864)
Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path (SPARK-36879)
Rebase timestamps in the session time zone saved in Parquet/Avro metadata (SPARK-37705)
Push down group by partition column for aggregate (SPARK-36646)
Aggregate (Min/Max/Count) push down for Parquet (SPARK-36645)
Parquet: enable matching schema columns by field id (SPARK-38094)
Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used (SPARK-37593)
Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support (SPARK-37974)
Support complex types for Parquet vectorized reader (SPARK-34863)
ORC
Remove check field name when reading/writing existing data in Orc (SPARK-37965)
Aggregate push down for ORC (SPARK-34960)
Support reading and writing ANSI intervals from/to ORC datasources (SPARK-36931)
Support number-only column names in ORC data sources (SPARK-36663)
JSON
Respect allowNonNumericNumbers when parsing quoted NaN and Infinity values in JSON reader (SPARK-38060)
Use CAST for datetime in CSV/JSON by default (SPARK-36536)
Align error message for unsupported key types in MapType in Json reader (SPARK-35320)
CSV
Fix referring to the corrupt record column from CSV (SPARK-38534)
null values should be saved as nothing instead of quoted empty Strings “” by default (SPARK-37575)
JDBC
Add the IMMEDIATE statement to the DB2 dialect truncate implementation (SPARK-30062)
Hive
Support writing Hive bucketed table (Hive file formats with Hive hash) (SPARK-32712)
Use expressions to filter Hive partitions at client side (SPARK-35437)
Support Dynamic Partition pruning for HiveTableScanExec (SPARK-36876)
InsertIntoHiveDir should use data source if it’s convertible (SPARK-38215)
Support writing Hive bucketed table (Parquet/ORC format with Hive hash) (SPARK-32709)
Node decommission
FallbackStorage shouldn’t attempt to resolve arbitrary “remote” hostname (SPARK-38062)
ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished (SPARK-38023)
Other notable changes
Add fine grained locking to BlockInfoManager (SPARK-37356)
Support mapping Spark gpu/fpga resource types to custom YARN resource type (SPARK-37208)
Report accurate shuffle block size if its skewed (SPARK-36967)
Supporting Netty Logging at the network layer (SPARK-36719)
Structured Streaming
Major features
Introduce Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches (SPARK-36533)
Other notable changes
Use StatefulOpClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)
Fix flatMapGroupsWithState timeout in batch with data for key (SPARK-38320)
Fix correctness issue on stream-stream outer join with RocksDB state store provider (SPARK-38684)
Support Trigger.AvailableNow on Kafka data source (SPARK-36649)
Optimize write path on RocksDB state store provider (SPARK-37224)
Introduce a new data source for providing a consistent set of rows per microbatch (SPARK-37062)
Use HashClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)
PySpark
Pandas API on Spark
Major improvements
‘distributed-sequence’ index optimization with being default (SPARK-36559, SPARK-36338)
Support to specify index type and name in pandas API on Spark (SPARK-36709)
Show default index type in SQL plans for pandas API on Spark (SPARK-38654)
Major features
Implement SparkSQL native ps.merge_asof (SPARK-36813)
Support TimedeltaIndex in pandas API on Spark (SPARK-37525)
Support Python’s timedelta (SPARK-37275, SPARK-37510)
Implement functions in CategoricalAccessor/CategoricalIndex (SPARK-36185)
Uses Python’s standard string formatter for SQL API in pandas API on Spark (SPARK-37436)
Support basic operations of timedelta Series/Index (SPARK-37510)
Support ps.MultiIndex.dtypes (SPARK-36930)
Implement Index.map (SPARK-36469)
Implement Series.__xor__ and Series.__rxor__ (SPARK-36653)
Implement unary operator
invert
of integral ps.Series/Index (SPARK-36003)Implement DataFrame.cov (SPARK-36396)
Support str and timestamp for (Series|DataFrame).describe() (SPARK-37657)
Support lambda
column
parameter ofDataFrame.rename
(SPARK-38763)
Other notable changes
Breaking changes
Drop references to Python 3.6 support in docs and python/docs (SPARK-36977)
Remove namedtuple hack by replacing built-in pickle to cloudpickle (SPARK-32079)
Bump minimum pandas version to 1.0.5 (SPARK-37465)
Major improvements
Provide a profiler for Python/Pandas UDFs (SPARK-37443)
Uses Python’s standard string formatter for SQL API in PySpark (SPARK-37516)
Expose SQL state and error class in PySpark exceptions (SPARK-36953)
Try to capture faulthanlder when a Python worker crashes (SPARK-36062)
Major features
Implement DataFrame.mapInArrow in Python (SPARK-37228)
Uses Python’s standard string formatter for SQL API in PySpark (SPARK-37516)
Add df.withMetadata pyspark API (SPARK-36642)
Support Python’s timedelta (SPARK-37275)
Expose tableExists in pyspark.sql.catalog (SPARK-36176)
Expose databaseExists in pyspark.sql.catalog (SPARK-36207)
Exposing functionExists in pyspark sql catalog (SPARK-36258)
Add Dataframe.observation to PySpark (SPARK-36263)
Add max_by/min_by API to PySpark (SPARK-36972)
Support to infer nested dict as a struct when creating a DataFrame (SPARK-35929)
Add bit/octet_length APIs to Scala, Python and R (SPARK-36751)
Support ILIKE API on Python (SPARK-36882)
Add isEmpty method for the Python DataFrame API (SPARK-37207)
Add multiple columns adding support (SPARK-35173)
Add SparkContext.addArchive in PySpark (SPARK-38278)
Make sql type reprs eval-able (SPARK-18621)
Inline type hints for fpm.py in python/pyspark/mllib (SPARK-37396)
Implement
dropna
parameter ofSeriesGroupBy.value_counts
(SPARK-38837)
MLLIB
Major features
Add distanceMeasure param to trainKMeansModel (SPARK-37118)
Expose LogisticRegression.setInitialModel, like KMeans et al do (SPARK-36481)
Support CrossValidatorModel get standard deviation of metrics for each paramMap (SPARK-36425)
Major improvements
Optimize some treeAggregates in MLlib by delaying allocations (SPARK-35848)
Rewrite _shared_params_code_gen.py to inline type hints for ml/param/shared.py (SPARK-37419)
Other notable changes
Update to breeze 1.2 (SPARK-35310)
SparkR
Migrate SparkR docs to pkgdown (SPARK-37474)
Expose make_date expression in R (SPARK-37108)
Add max_by/min_by API to SparkR (SPARK-36976)
Support ILIKE API on R (SPARK-36899)
Add sec and csc as R functions (SPARK-36824)
Add bit/octet_length APIs to Scala, Python and R (SPARK-36751)
Add cot as an R function (SPARK-36688)
UI
Speculation metrics summary at stage level (SPARK-36038)
Unified shuffle read block time to shuffle read fetch wait time in StagePage (SPARK-37469)
Add modified configs for SQL execution in UI (SPARK-34735)
Make ThriftServer recognize spark.sql.redaction.string.regex (SPARK-36400)
Attach and start handler after application started in UI (SPARK-36237)
Add commit duration to SQL tab’s graph node (SPARK-34399)
Support RocksDB backend in Spark History Server (SPARK-37680)
Show options for Pandas API on Spark in UI (SPARK-38656)
Rename ‘SQL’ to ‘SQL / DataFrame’ in SQL UI page (SPARK-38657)
Build
Migrating from log4j 1 to log4j 2 (SPARK-37814)
Upgrade log4j2 to 2.17.2 (SPARK-38544)
Upgrade to Py4J 0.10.9.5 (SPARK-38563)
Update ORC to 1.7.4 (SPARK-38866)
Update datatables to 1.10.25 (SPARK-38924)
System environment
Operating System: Ubuntu 20.04.4 LTS
Java: Zulu 8.56.0.21-CA-linux64
Scala: 2.12.14
Python: 3.9.5
R: 4.1.3
Delta Lake: 1.2.1
Installed Python libraries
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
Antergos Linux |
2015.10 (ISO-Rolling) |
argon2-cffi |
20.1.0 |
async-generator |
1.10 |
attrs |
21.2.0 |
backcall |
0.2.0 |
backports.entry-points-selectable |
1.1.1 |
bleach |
4.0.0 |
boto3 |
1.21.18 |
botocore |
1.24.18 |
certifi |
2021.10.8 |
cffi |
1.14.6 |
chardet |
4.0.0 |
charset-normalizer |
2.0.4 |
cycler |
0.10.0 |
Cython |
0.29.24 |
dbus-python |
1.2.16 |
debugpy |
1.4.1 |
decorator |
5.1.0 |
defusedxml |
0.7.1 |
distlib |
0.3.4 |
distro-info |
0.23ubuntu1 |
entrypoints |
0.3 |
facets-overview |
1.0.0 |
filelock |
3.6.0 |
idna |
3.2 |
ipykernel |
6.12.1 |
ipython |
7.32.0 |
ipython-genutils |
0.2.0 |
ipywidgets |
7.7.0 |
jedi |
0.18.0 |
Jinja2 |
2.11.3 |
jmespath |
0.10.0 |
joblib |
1.0.1 |
jsonschema |
3.2.0 |
jupyter-client |
6.1.12 |
jupyter-core |
4.8.1 |
jupyterlab-pygments |
0.1.2 |
jupyterlab-widgets |
1.0.0 |
kiwisolver |
1.3.1 |
MarkupSafe |
2.0.1 |
matplotlib |
3.4.3 |
matplotlib-inline |
0.1.2 |
mistune |
0.8.4 |
nbclient |
0.5.3 |
nbconvert |
6.1.0 |
nbformat |
5.1.3 |
nest-asyncio |
1.5.1 |
notebook |
6.4.5 |
numpy |
1.20.3 |
packaging |
21.0 |
pandas |
1.3.4 |
pandocfilters |
1.4.3 |
parso |
0.8.2 |
patsy |
0.5.2 |
pexpect |
4.8.0 |
pickleshare |
0.7.5 |
Pillow |
8.4.0 |
pip |
21.2.4 |
platformdirs |
2.5.2 |
plotly |
5.6.0 |
prometheus-client |
0.11.0 |
prompt-toolkit |
3.0.20 |
protobuf |
3.20.1 |
psutil |
5.8.0 |
psycopg2 |
2.9.3 |
ptyprocess |
0.7.0 |
pyarrow |
7.0.0 |
pycparser |
2.20 |
Pygments |
2.10.0 |
PyGObject |
3.36.0 |
pyodbc |
4.0.31 |
pyparsing |
3.0.4 |
pyrsistent |
0.18.0 |
python-apt |
2.0.0+ubuntu0.20.4.7 |
python-dateutil |
2.8.2 |
pytz |
2021.3 |
pyzmq |
22.2.1 |
requests |
2.26.0 |
requests-unixsocket |
0.2.0 |
s3transfer |
0.5.2 |
scikit-learn |
0.24.2 |
scipy |
1.7.1 |
seaborn |
0.11.2 |
Send2Trash |
1.8.0 |
setuptools |
58.0.4 |
six |
1.16.0 |
ssh-import-id |
5.10 |
statsmodels |
0.12.2 |
tenacity |
8.0.1 |
terminado |
0.9.4 |
testpath |
0.5.0 |
threadpoolctl |
2.2.0 |
tornado |
6.1 |
traitlets |
5.1.0 |
unattended-upgrades |
0.1 |
urllib3 |
1.26.7 |
virtualenv |
20.8.0 |
wcwidth |
0.2.5 |
webencodings |
0.5.1 |
wheel |
0.37.0 |
widgetsnbextension |
3.6.0 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2022-05-06.
Library |
Version |
Library |
Version |
Library |
Version |
---|---|---|---|---|---|
askpass |
1.1 |
assertthat |
0.2.1 |
backports |
1.4.1 |
base |
4.1.3 |
base64enc |
0.1-3 |
bit |
4.0.4 |
bit64 |
4.0.5 |
blob |
1.2.3 |
boot |
1.3-28 |
brew |
1.0-7 |
brio |
1.1.3 |
broom |
0.8.0 |
bslib |
0.3.1 |
cachem |
1.0.6 |
callr |
3.7.0 |
caret |
6.0-92 |
cellranger |
1.1.0 |
chron |
2.3-56 |
class |
7.3-20 |
cli |
3.3.0 |
clipr |
0.8.0 |
cluster |
2.1.3 |
codetools |
0.2-18 |
colorspace |
2.0-3 |
commonmark |
1.8.0 |
compiler |
4.1.3 |
config |
0.3.1 |
cpp11 |
0.4.2 |
crayon |
1.5.1 |
credentials |
1.3.2 |
curl |
4.3.2 |
data.table |
1.14.2 |
datasets |
4.1.3 |
DBI |
1.1.2 |
dbplyr |
2.1.1 |
desc |
1.4.1 |
devtools |
2.4.3 |
diffobj |
0.3.5 |
digest |
0.6.29 |
dplyr |
1.0.9 |
dtplyr |
1.2.1 |
e1071 |
1.7-9 |
ellipsis |
0.3.2 |
evaluate |
0.15 |
fansi |
1.0.3 |
farver |
2.1.0 |
fastmap |
1.1.0 |
fontawesome |
0.2.2 |
forcats |
0.5.1 |
foreach |
1.5.2 |
foreign |
0.8-82 |
forge |
0.2.0 |
fs |
1.5.2 |
future |
1.25.0 |
future.apply |
1.9.0 |
gargle |
1.2.0 |
generics |
0.1.2 |
gert |
1.6.0 |
ggplot2 |
3.3.6 |
gh |
1.3.0 |
gitcreds |
0.1.1 |
glmnet |
4.1-4 |
globals |
0.14.0 |
glue |
1.6.2 |
googledrive |
2.0.0 |
googlesheets4 |
1.0.0 |
gower |
1.0.0 |
graphics |
4.1.3 |
grDevices |
4.1.3 |
grid |
4.1.3 |
gridExtra |
2.3 |
gsubfn |
0.7 |
gtable |
0.3.0 |
hardhat |
0.2.0 |
haven |
2.5.0 |
highr |
0.9 |
hms |
1.1.1 |
htmltools |
0.5.2 |
htmlwidgets |
1.5.4 |
httpuv |
1.6.5 |
httr |
1.4.3 |
ids |
1.0.1 |
ini |
0.3.1 |
ipred |
0.9-12 |
isoband |
0.2.5 |
iterators |
1.0.14 |
jquerylib |
0.1.4 |
jsonlite |
1.8.0 |
KernSmooth |
2.23-20 |
knitr |
1.39 |
labeling |
0.4.2 |
later |
1.3.0 |
lattice |
0.20-45 |
lava |
1.6.10 |
lifecycle |
1.0.1 |
listenv |
0.8.0 |
lubridate |
1.8.0 |
magrittr |
2.0.3 |
markdown |
1.1 |
MASS |
7.3-56 |
Matrix |
1.4-1 |
memoise |
2.0.1 |
methods |
4.1.3 |
mgcv |
1.8-40 |
mime |
0.12 |
ModelMetrics |
1.2.2.2 |
modelr |
0.1.8 |
munsell |
0.5.0 |
nlme |
3.1-157 |
nnet |
7.3-17 |
numDeriv |
2016.8-1.1 |
openssl |
2.0.0 |
parallel |
4.1.3 |
parallelly |
1.31.1 |
pillar |
1.7.0 |
pkgbuild |
1.3.1 |
pkgconfig |
2.0.3 |
pkgload |
1.2.4 |
plogr |
0.2.0 |
plyr |
1.8.7 |
praise |
1.0.0 |
prettyunits |
1.1.1 |
pROC |
1.18.0 |
processx |
3.5.3 |
prodlim |
2019.11.13 |
progress |
1.2.2 |
progressr |
0.10.0 |
promises |
1.2.0.1 |
proto |
1.0.0 |
proxy |
0.4-26 |
ps |
1.7.0 |
purrr |
0.3.4 |
r2d3 |
0.2.6 |
R6 |
2.5.1 |
randomForest |
4.7-1 |
rappdirs |
0.3.3 |
rcmdcheck |
1.4.0 |
RColorBrewer |
1.1-3 |
Rcpp |
1.0.8.3 |
RcppEigen |
0.3.3.9.2 |
readr |
2.1.2 |
readxl |
1.4.0 |
recipes |
0.2.0 |
rematch |
1.0.1 |
rematch2 |
2.1.2 |
remotes |
2.4.2 |
reprex |
2.0.1 |
reshape2 |
1.4.4 |
rlang |
1.0.2 |
rmarkdown |
2.14 |
RODBC |
1.3-19 |
roxygen2 |
7.1.2 |
rpart |
4.1.16 |
rprojroot |
2.0.3 |
Rserve |
1.8-10 |
RSQLite |
2.2.13 |
rstudioapi |
0.13 |
rversions |
2.1.1 |
rvest |
1.0.2 |
sass |
0.4.1 |
scales |
1.2.0 |
selectr |
0.4-2 |
sessioninfo |
1.2.2 |
shape |
1.4.6 |
shiny |
1.7.1 |
sourcetools |
0.1.7 |
sparklyr |
1.7.5 |
SparkR |
3.3.0 |
spatial |
7.3-11 |
splines |
4.1.3 |
sqldf |
0.4-11 |
SQUAREM |
2021.1 |
stats |
4.1.3 |
stats4 |
4.1.3 |
stringi |
1.7.6 |
stringr |
1.4.0 |
survival |
3.2-13 |
sys |
3.4 |
tcltk |
4.1.3 |
testthat |
3.1.4 |
tibble |
3.1.7 |
tidyr |
1.2.0 |
tidyselect |
1.1.2 |
tidyverse |
1.3.1 |
timeDate |
3043.102 |
tinytex |
0.38 |
tools |
4.1.3 |
tzdb |
0.3.0 |
usethis |
2.1.5 |
utf8 |
1.2.2 |
utils |
4.1.3 |
uuid |
1.1-0 |
vctrs |
0.4.1 |
viridisLite |
0.4.0 |
vroom |
1.5.7 |
waldo |
0.4.0 |
whisker |
0.4 |
withr |
2.5.0 |
xfun |
0.30 |
xml2 |
1.3.3 |
xopen |
1.0.0 |
xtable |
1.8-4 |
yaml |
2.3.5 |
zip |
2.2.0 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
Group ID |
Artifact ID |
Version |
---|---|---|
antlr |
antlr |
2.7.7 |
com.amazonaws |
amazon-kinesis-client |
1.12.0 |
com.amazonaws |
aws-java-sdk-autoscaling |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudformation |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudfront |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudhsm |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudsearch |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudtrail |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudwatch |
1.12.189 |
com.amazonaws |
aws-java-sdk-cloudwatchmetrics |
1.12.189 |
com.amazonaws |
aws-java-sdk-codedeploy |
1.12.189 |
com.amazonaws |
aws-java-sdk-cognitoidentity |
1.12.189 |
com.amazonaws |
aws-java-sdk-cognitosync |
1.12.189 |
com.amazonaws |
aws-java-sdk-config |
1.12.189 |
com.amazonaws |
aws-java-sdk-core |
1.12.189 |
com.amazonaws |
aws-java-sdk-datapipeline |
1.12.189 |
com.amazonaws |
aws-java-sdk-directconnect |
1.12.189 |
com.amazonaws |
aws-java-sdk-directory |
1.12.189 |
com.amazonaws |
aws-java-sdk-dynamodb |
1.12.189 |
com.amazonaws |
aws-java-sdk-ec2 |
1.12.189 |
com.amazonaws |
aws-java-sdk-ecs |
1.12.189 |
com.amazonaws |
aws-java-sdk-efs |
1.12.189 |
com.amazonaws |
aws-java-sdk-elasticache |
1.12.189 |
com.amazonaws |
aws-java-sdk-elasticbeanstalk |
1.12.189 |
com.amazonaws |
aws-java-sdk-elasticloadbalancing |
1.12.189 |
com.amazonaws |
aws-java-sdk-elastictranscoder |
1.12.189 |
com.amazonaws |
aws-java-sdk-emr |
1.12.189 |
com.amazonaws |
aws-java-sdk-glacier |
1.12.189 |
com.amazonaws |
aws-java-sdk-glue |
1.12.189 |
com.amazonaws |
aws-java-sdk-iam |
1.12.189 |
com.amazonaws |
aws-java-sdk-importexport |
1.12.189 |
com.amazonaws |
aws-java-sdk-kinesis |
1.12.189 |
com.amazonaws |
aws-java-sdk-kms |
1.12.189 |
com.amazonaws |
aws-java-sdk-lambda |
1.12.189 |
com.amazonaws |
aws-java-sdk-logs |
1.12.189 |
com.amazonaws |
aws-java-sdk-machinelearning |
1.12.189 |
com.amazonaws |
aws-java-sdk-opsworks |
1.12.189 |
com.amazonaws |
aws-java-sdk-rds |
1.12.189 |
com.amazonaws |
aws-java-sdk-redshift |
1.12.189 |
com.amazonaws |
aws-java-sdk-route53 |
1.12.189 |
com.amazonaws |
aws-java-sdk-s3 |
1.12.189 |
com.amazonaws |
aws-java-sdk-ses |
1.12.189 |
com.amazonaws |
aws-java-sdk-simpledb |
1.12.189 |
com.amazonaws |
aws-java-sdk-simpleworkflow |
1.12.189 |
com.amazonaws |
aws-java-sdk-sns |
1.12.189 |
com.amazonaws |
aws-java-sdk-sqs |
1.12.189 |
com.amazonaws |
aws-java-sdk-ssm |
1.12.189 |
com.amazonaws |
aws-java-sdk-storagegateway |
1.12.189 |
com.amazonaws |
aws-java-sdk-sts |
1.12.189 |
com.amazonaws |
aws-java-sdk-support |
1.12.189 |
com.amazonaws |
aws-java-sdk-swf-libraries |
1.11.22 |
com.amazonaws |
aws-java-sdk-workspaces |
1.12.189 |
com.amazonaws |
jmespath-java |
1.12.189 |
com.chuusai |
shapeless_2.12 |
2.3.3 |
com.clearspring.analytics |
stream |
2.9.6 |
com.databricks |
Rserve |
1.8-3 |
com.databricks |
jets3t |
0.7.1-0 |
com.databricks.scalapb |
compilerplugin_2.12 |
0.4.15-10 |
com.databricks.scalapb |
scalapb-runtime_2.12 |
0.4.15-10 |
com.esotericsoftware |
kryo-shaded |
4.0.2 |
com.esotericsoftware |
minlog |
1.3.0 |
com.fasterxml |
classmate |
1.3.4 |
com.fasterxml.jackson.core |
jackson-annotations |
2.13.3 |
com.fasterxml.jackson.core |
jackson-core |
2.13.3 |
com.fasterxml.jackson.core |
jackson-databind |
2.13.3 |
com.fasterxml.jackson.dataformat |
jackson-dataformat-cbor |
2.13.3 |
com.fasterxml.jackson.datatype |
jackson-datatype-joda |
2.13.3 |
com.fasterxml.jackson.module |
jackson-module-paranamer |
2.13.3 |
com.fasterxml.jackson.module |
jackson-module-scala_2.12 |
2.13.3 |
com.github.ben-manes.caffeine |
caffeine |
2.3.4 |
com.github.fommil |
jniloader |
1.1 |
com.github.fommil.netlib |
core |
1.1.2 |
com.github.fommil.netlib |
native_ref-java |
1.1 |
com.github.fommil.netlib |
native_ref-java-natives |
1.1 |
com.github.fommil.netlib |
native_system-java |
1.1 |
com.github.fommil.netlib |
native_system-java-natives |
1.1 |
com.github.fommil.netlib |
netlib-native_ref-linux-x86_64-natives |
1.1 |
com.github.fommil.netlib |
netlib-native_system-linux-x86_64-natives |
1.1 |
com.github.luben |
zstd-jni |
1.5.2-1 |
com.github.wendykierp |
JTransforms |
3.1 |
com.google.code.findbugs |
jsr305 |
3.0.0 |
com.google.code.gson |
gson |
2.8.6 |
com.google.crypto.tink |
tink |
1.6.1 |
com.google.flatbuffers |
flatbuffers-java |
1.12.0 |
com.google.guava |
guava |
15.0 |
com.google.protobuf |
protobuf-java |
2.6.1 |
com.h2database |
h2 |
2.0.204 |
com.helger |
profiler |
1.1.1 |
com.jcraft |
jsch |
0.1.50 |
com.jolbox |
bonecp |
0.8.0.RELEASE |
com.lihaoyi |
sourcecode_2.12 |
0.1.9 |
com.microsoft.azure |
azure-data-lake-store-sdk |
2.3.9 |
com.ning |
compress-lzf |
1.1 |
com.sun.mail |
javax.mail |
1.5.2 |
com.tdunning |
json |
1.8 |
com.thoughtworks.paranamer |
paranamer |
2.8 |
com.trueaccord.lenses |
lenses_2.12 |
0.4.12 |
com.twitter |
chill-java |
0.10.0 |
com.twitter |
chill_2.12 |
0.10.0 |
com.twitter |
util-app_2.12 |
7.1.0 |
com.twitter |
util-core_2.12 |
7.1.0 |
com.twitter |
util-function_2.12 |
7.1.0 |
com.twitter |
util-jvm_2.12 |
7.1.0 |
com.twitter |
util-lint_2.12 |
7.1.0 |
com.twitter |
util-registry_2.12 |
7.1.0 |
com.twitter |
util-stats_2.12 |
7.1.0 |
com.typesafe |
config |
1.2.1 |
com.typesafe.scala-logging |
scala-logging_2.12 |
3.7.2 |
com.univocity |
univocity-parsers |
2.9.1 |
com.zaxxer |
HikariCP |
4.0.3 |
commons-cli |
commons-cli |
1.5.0 |
commons-codec |
commons-codec |
1.15 |
commons-collections |
commons-collections |
3.2.2 |
commons-dbcp |
commons-dbcp |
1.4 |
commons-fileupload |
commons-fileupload |
1.3.3 |
commons-httpclient |
commons-httpclient |
3.1 |
commons-io |
commons-io |
2.11.0 |
commons-lang |
commons-lang |
2.6 |
commons-logging |
commons-logging |
1.1.3 |
commons-pool |
commons-pool |
1.5.4 |
dev.ludovic.netlib |
arpack |
2.2.1 |
dev.ludovic.netlib |
blas |
2.2.1 |
dev.ludovic.netlib |
lapack |
2.2.1 |
hive-2.3__hadoop-3.2 |
jets3t-0.7 |
liball_deps_2.12 |
info.ganglia.gmetric4j |
gmetric4j |
1.0.10 |
io.airlift |
aircompressor |
0.21 |
io.delta |
delta-sharing-spark_2.12 |
0.4.0 |
io.dropwizard.metrics |
metrics-core |
4.1.1 |
io.dropwizard.metrics |
metrics-graphite |
4.1.1 |
io.dropwizard.metrics |
metrics-healthchecks |
4.1.1 |
io.dropwizard.metrics |
metrics-jetty9 |
4.1.1 |
io.dropwizard.metrics |
metrics-jmx |
4.1.1 |
io.dropwizard.metrics |
metrics-json |
4.1.1 |
io.dropwizard.metrics |
metrics-jvm |
4.1.1 |
io.dropwizard.metrics |
metrics-servlets |
4.1.1 |
io.netty |
netty-all |
4.1.74.Final |
io.netty |
netty-buffer |
4.1.74.Final |
io.netty |
netty-codec |
4.1.74.Final |
io.netty |
netty-common |
4.1.74.Final |
io.netty |
netty-handler |
4.1.74.Final |
io.netty |
netty-resolver |
4.1.74.Final |
io.netty |
netty-tcnative-classes |
2.0.48.Final |
io.netty |
netty-transport |
4.1.74.Final |
io.netty |
netty-transport-classes-epoll |
4.1.74.Final |
io.netty |
netty-transport-classes-kqueue |
4.1.74.Final |
io.netty |
netty-transport-native-epoll-linux-aarch_64 |
4.1.74.Final |
io.netty |
netty-transport-native-epoll-linux-x86_64 |
4.1.74.Final |
io.netty |
netty-transport-native-kqueue-osx-aarch_64 |
4.1.74.Final |
io.netty |
netty-transport-native-kqueue-osx-x86_64 |
4.1.74.Final |
io.netty |
netty-transport-native-unix-common |
4.1.74.Final |
io.prometheus |
simpleclient |
0.7.0 |
io.prometheus |
simpleclient_common |
0.7.0 |
io.prometheus |
simpleclient_dropwizard |
0.7.0 |
io.prometheus |
simpleclient_pushgateway |
0.7.0 |
io.prometheus |
simpleclient_servlet |
0.7.0 |
io.prometheus.jmx |
collector |
0.12.0 |
jakarta.annotation |
jakarta.annotation-api |
1.3.5 |
jakarta.servlet |
jakarta.servlet-api |
4.0.3 |
jakarta.validation |
jakarta.validation-api |
2.0.2 |
jakarta.ws.rs |
jakarta.ws.rs-api |
2.1.6 |
javax.activation |
activation |
1.1.1 |
javax.annotation |
javax.annotation-api |
1.3.2 |
javax.el |
javax.el-api |
2.2.4 |
javax.jdo |
jdo-api |
3.0.1 |
javax.transaction |
jta |
1.1 |
javax.transaction |
transaction-api |
1.1 |
javax.xml.bind |
jaxb-api |
2.2.11 |
javolution |
javolution |
5.5.1 |
jline |
jline |
2.14.6 |
joda-time |
joda-time |
2.10.13 |
maven-trees |
hive-2.3__hadoop-3.2 |
liball_deps_2.12 |
net.java.dev.jna |
jna |
5.8.0 |
net.razorvine |
pickle |
1.2 |
net.sf.jpam |
jpam |
1.1 |
net.sf.opencsv |
opencsv |
2.3 |
net.sf.supercsv |
super-csv |
2.2.0 |
net.snowflake |
snowflake-ingest-sdk |
0.9.6 |
net.snowflake |
snowflake-jdbc |
3.13.14 |
net.snowflake |
spark-snowflake_2.12 |
2.10.0-spark_3.1 |
net.sourceforge.f2j |
arpack_combined_all |
0.1 |
org.acplt.remotetea |
remotetea-oncrpc |
1.1.2 |
org.antlr |
ST4 |
4.0.4 |
org.antlr |
antlr-runtime |
3.5.2 |
org.antlr |
antlr4-runtime |
4.8 |
org.antlr |
stringtemplate |
3.2.1 |
org.apache.ant |
ant |
1.9.2 |
org.apache.ant |
ant-jsch |
1.9.2 |
org.apache.ant |
ant-launcher |
1.9.2 |
org.apache.arrow |
arrow-format |
7.0.0 |
org.apache.arrow |
arrow-memory-core |
7.0.0 |
org.apache.arrow |
arrow-memory-netty |
7.0.0 |
org.apache.arrow |
arrow-vector |
7.0.0 |
org.apache.avro |
avro |
1.11.0 |
org.apache.avro |
avro-ipc |
1.11.0 |
org.apache.avro |
avro-mapred |
1.11.0 |
org.apache.commons |
commons-collections4 |
4.4 |
org.apache.commons |
commons-compress |
1.21 |
org.apache.commons |
commons-crypto |
1.1.0 |
org.apache.commons |
commons-lang3 |
3.12.0 |
org.apache.commons |
commons-math3 |
3.6.1 |
org.apache.commons |
commons-text |
1.9 |
org.apache.curator |
curator-client |
2.13.0 |
org.apache.curator |
curator-framework |
2.13.0 |
org.apache.curator |
curator-recipes |
2.13.0 |
org.apache.derby |
derby |
10.14.2.0 |
org.apache.hadoop |
hadoop-client-api |
3.3.2-databricks |
org.apache.hadoop |
hadoop-client-runtime |
3.3.2 |
org.apache.hive |
hive-beeline |
2.3.9 |
org.apache.hive |
hive-cli |
2.3.9 |
org.apache.hive |
hive-jdbc |
2.3.9 |
org.apache.hive |
hive-llap-client |
2.3.9 |
org.apache.hive |
hive-llap-common |
2.3.9 |
org.apache.hive |
hive-serde |
2.3.9 |
org.apache.hive |
hive-shims |
2.3.9 |
org.apache.hive |
hive-storage-api |
2.7.2 |
org.apache.hive.shims |
hive-shims-0.23 |
2.3.9 |
org.apache.hive.shims |
hive-shims-common |
2.3.9 |
org.apache.hive.shims |
hive-shims-scheduler |
2.3.9 |
org.apache.httpcomponents |
httpclient |
4.5.13 |
org.apache.httpcomponents |
httpcore |
4.4.14 |
org.apache.ivy |
ivy |
2.5.0 |
org.apache.logging.log4j |
log4j-1.2-api |
2.17.2 |
org.apache.logging.log4j |
log4j-api |
2.17.2 |
org.apache.logging.log4j |
log4j-core |
2.17.2 |
org.apache.logging.log4j |
log4j-slf4j-impl |
2.17.2 |
org.apache.mesos |
mesos-shaded-protobuf |
1.4.0 |
org.apache.orc |
orc-core |
1.7.4 |
org.apache.orc |
orc-mapreduce |
1.7.4 |
org.apache.orc |
orc-shims |
1.7.4 |
org.apache.parquet |
parquet-column |
1.12.0-databricks-0004 |
org.apache.parquet |
parquet-common |
1.12.0-databricks-0004 |
org.apache.parquet |
parquet-encoding |
1.12.0-databricks-0004 |
org.apache.parquet |
parquet-format-structures |
1.12.0-databricks-0004 |
org.apache.parquet |
parquet-hadoop |
1.12.0-databricks-0004 |
org.apache.parquet |
parquet-jackson |
1.12.0-databricks-0004 |
org.apache.thrift |
libfb303 |
0.9.3 |
org.apache.thrift |
libthrift |
0.12.0 |
org.apache.xbean |
xbean-asm9-shaded |
4.20 |
org.apache.yetus |
audience-annotations |
0.5.0 |
org.apache.zookeeper |
zookeeper |
3.6.2 |
org.apache.zookeeper |
zookeeper-jute |
3.6.2 |
org.checkerframework |
checker-qual |
3.5.0 |
org.codehaus.jackson |
jackson-core-asl |
1.9.13 |
org.codehaus.jackson |
jackson-mapper-asl |
1.9.13 |
org.codehaus.janino |
commons-compiler |
3.0.16 |
org.codehaus.janino |
janino |
3.0.16 |
org.datanucleus |
datanucleus-api-jdo |
4.2.4 |
org.datanucleus |
datanucleus-core |
4.1.17 |
org.datanucleus |
datanucleus-rdbms |
4.1.19 |
org.datanucleus |
javax.jdo |
3.2.0-m3 |
org.eclipse.jetty |
jetty-client |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-continuation |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-http |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-io |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-jndi |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-plus |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-proxy |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-security |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-server |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-servlet |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-servlets |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-util |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-util-ajax |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-webapp |
9.4.46.v20220331 |
org.eclipse.jetty |
jetty-xml |
9.4.46.v20220331 |
org.eclipse.jetty.websocket |
websocket-api |
9.4.46.v20220331 |
org.eclipse.jetty.websocket |
websocket-client |
9.4.46.v20220331 |
org.eclipse.jetty.websocket |
websocket-common |
9.4.46.v20220331 |
org.eclipse.jetty.websocket |
websocket-server |
9.4.46.v20220331 |
org.eclipse.jetty.websocket |
websocket-servlet |
9.4.46.v20220331 |
org.fusesource.leveldbjni |
leveldbjni-all |
1.8 |
org.glassfish.hk2 |
hk2-api |
2.6.1 |
org.glassfish.hk2 |
hk2-locator |
2.6.1 |
org.glassfish.hk2 |
hk2-utils |
2.6.1 |
org.glassfish.hk2 |
osgi-resource-locator |
1.0.3 |
org.glassfish.hk2.external |
aopalliance-repackaged |
2.6.1 |
org.glassfish.hk2.external |
jakarta.inject |
2.6.1 |
org.glassfish.jersey.containers |
jersey-container-servlet |
2.34 |
org.glassfish.jersey.containers |
jersey-container-servlet-core |
2.34 |
org.glassfish.jersey.core |
jersey-client |
2.34 |
org.glassfish.jersey.core |
jersey-common |
2.34 |
org.glassfish.jersey.core |
jersey-server |
2.34 |
org.glassfish.jersey.inject |
jersey-hk2 |
2.34 |
org.hibernate.validator |
hibernate-validator |
6.1.0.Final |
org.javassist |
javassist |
3.25.0-GA |
org.jboss.logging |
jboss-logging |
3.3.2.Final |
org.jdbi |
jdbi |
2.63.1 |
org.jetbrains |
annotations |
17.0.0 |
org.joda |
joda-convert |
1.7 |
org.jodd |
jodd-core |
3.5.2 |
org.json4s |
json4s-ast_2.12 |
3.7.0-M11 |
org.json4s |
json4s-core_2.12 |
3.7.0-M11 |
org.json4s |
json4s-jackson_2.12 |
3.7.0-M11 |
org.json4s |
json4s-scalap_2.12 |
3.7.0-M11 |
org.lz4 |
lz4-java |
1.8.0 |
org.mariadb.jdbc |
mariadb-java-client |
2.7.4 |
org.objenesis |
objenesis |
2.5.1 |
org.postgresql |
postgresql |
42.3.3 |
org.roaringbitmap |
RoaringBitmap |
0.9.25 |
org.roaringbitmap |
shims |
0.9.25 |
org.rocksdb |
rocksdbjni |
6.24.2 |
org.rosuda.REngine |
REngine |
2.1.0 |
org.scala-lang |
scala-compiler_2.12 |
2.12.14 |
org.scala-lang |
scala-library_2.12 |
2.12.14 |
org.scala-lang |
scala-reflect_2.12 |
2.12.14 |
org.scala-lang.modules |
scala-collection-compat_2.12 |
2.4.3 |
org.scala-lang.modules |
scala-parser-combinators_2.12 |
1.1.2 |
org.scala-lang.modules |
scala-xml_2.12 |
1.2.0 |
org.scala-sbt |
test-interface |
1.0 |
org.scalacheck |
scalacheck_2.12 |
1.14.2 |
org.scalactic |
scalactic_2.12 |
3.0.8 |
org.scalanlp |
breeze-macros_2.12 |
1.2 |
org.scalanlp |
breeze_2.12 |
1.2 |
org.scalatest |
scalatest_2.12 |
3.0.8 |
org.slf4j |
jcl-over-slf4j |
1.7.36 |
org.slf4j |
jul-to-slf4j |
1.7.36 |
org.slf4j |
slf4j-api |
1.7.36 |
org.spark-project.spark |
unused |
1.0.0 |
org.threeten |
threeten-extra |
1.5.0 |
org.tukaani |
xz |
1.8 |
org.typelevel |
algebra_2.12 |
2.0.1 |
org.typelevel |
cats-kernel_2.12 |
2.1.1 |
org.typelevel |
macro-compat_2.12 |
1.1.1 |
org.typelevel |
spire-macros_2.12 |
0.17.0 |
org.typelevel |
spire-platform_2.12 |
0.17.0 |
org.typelevel |
spire-util_2.12 |
0.17.0 |
org.typelevel |
spire_2.12 |
0.17.0 |
org.wildfly.openssl |
wildfly-openssl |
1.0.7.Final |
org.xerial |
sqlite-jdbc |
3.8.11.2 |
org.xerial.snappy |
snappy-java |
1.1.8.4 |
org.yaml |
snakeyaml |
1.24 |
oro |
oro |
2.0.8 |
pl.edu.icm |
JLargeArrays |
1.5 |
software.amazon.ion |
ion-java |
1.0.2 |
stax |
stax-api |
1.0.1 |