メインコンテンツまでスキップ
に最終更新

Databricks Runtime 18.0 (Beta)

The following release notes provide information about Databricks Runtime 18.0 (Beta), powered by Apache Spark 4.0.0.

Databricks released this version in December 2025.

Beta

Databricks Runtime 18.0 is in Beta. The contents of the supported environments might change during the Beta. Changes can include the list of packages or versions of installed packages.

New features and improvements​

JDK 21 is generally available​

Databricks Runtime 18.0 uses JDK 21 as the default Java Development Kit. JDK 21 is a long-term support (LTS) release. JDK 17 is available as a fallback option.

For information about configuring JDK versions for your clusters, see Create a cluster with a specific JDK version.

Redshift JDBC driver upgraded to 2.1.0.28​

The Redshift JDBC driver has been upgraded to version 2.1.0.28.

SQL window functions in metric views​

You can now use SQL window functions in metric views to calculate running totals, rankings, and other window-based calculations.

Graceful shutdown for Spark applications​

Spark applications now support graceful shutdown, allowing in-progress tasks to complete before the application terminates.

Dynamic shuffle partition adjustment in stateless streaming queries​

You can now change the number of shuffle partitions in stateless streaming queries without restarting the query.

Adaptive Query Execution and auto-optimized shuffle in stateless streaming queries​

Adaptive Query Execution (AQE) and auto-optimized shuffle (AOS) are now supported in stateless streaming queries.

FILTER clause for measure aggregate functions in metric views​

You can now use the FILTER clause with measure aggregate functions in metric views to define per-aggregate filters when referencing metric view measures.

Apache Parquet library upgraded to 1.16.0​

The Apache Parquet library has been upgraded to version 1.16.0.

Behavioral changes​

FSCK REPAIR TABLE includes metadata repair by default​

The FSCK REPAIR TABLE command now includes an initial metadata repair step before checking for missing data files. The command can work on tables with corrupt checkpoints or invalid partition values.

Nullability preserved for array and map types in Spark Connect Scala client​

The nullability of array and map types is now preserved for typed literals in the Spark Connect Scala client. Previously, elements of arrays and values of maps were always nullable.

FSCK REPAIR TABLE DRY RUN output schema updated​

The dataFilePath column in the FSCK REPAIR TABLE DRY RUN output schema is now nullable to support reporting new issue types where the data file path isn't applicable.

SHOW TABLES DROPPED respects LIMIT clause​

The SHOW TABLES DROPPED command now correctly respects the LIMIT clause.

Proration factors aligned between reads and auto-optimized writes​

Proration factors for partition sizing now use fractional values consistently across read operations and auto-optimized writes. This change might result in a different number of tasks for read operations.

Python UDF execution unified across PySpark and Unity Catalog​

Databricks now uses a unified Arrow-based execution framework for both session-based (PySpark) and Unity Catalog Python UDFs, removing the five-UDF limit for Unity Catalog workloads. Arrow is now enabled by default, which might introduce breaking changes in type coercion when Python return values don't match the declared schema. You can opt out by setting the appropriate SQL configuration or annotating UDFs with @udf(useArrow=False).

Improved error messages for Kafka connector login module issues​

When using the Kafka connector with an unshaded login module class, Databricks now provides error messages that suggest using the correct shaded class prefix (kafkashaded.org.apache.kafka or kafkashaded.software.amazon.msk.auth.iam).

Time travel restrictions and VACUUM retention behavior​

Databricks now blocks time travel queries beyond the deletedFileRetentionDuration threshold for all tables. The VACUUM command ignores the retention duration argument except when the value is 0 hours. You can't set deletedFileRetentionDuration larger than logRetentionDuration or vice versa.

BinaryType maps to bytes by default in PySpark​

BinaryType now consistently maps to Python bytes in PySpark. Previously, PySpark mapped BinaryType to either bytes or bytearray depending on the context. To restore the old behavior, set spark.sql.execution.pyspark.binaryAsBytes to false.

Library upgrades​

Upgraded Python libraries:

  • anyio from 4.6.2 to 4.7.0
  • asttokens from 2.0.5 to 3.0.0
  • azure-core from 1.34.0 to 1.36.0
  • azure-mgmt-core from 1.5.0 to 1.6.0
  • azure-storage-blob from 12.23.0 to 12.27.1
  • azure-storage-file-datalake from 12.17.0 to 12.22.0
  • boto3 from 1.36.2 to 1.40.45
  • botocore from 1.36.3 to 1.40.45
  • certifi from 2025.1.31 to 2025.4.26
  • click from 8.1.7 to 8.1.8
  • cryptography from 43.0.3 to 44.0.1
  • Cython from 3.0.12 to 3.1.5
  • databricks-sdk from 0.49.0 to 0.67.0
  • Deprecated from 1.2.13 to 1.2.18
  • executing from 0.8.3 to 1.2.0
  • fastapi from 0.115.12 to 0.121.0
  • google-api-core from 2.20.0 to 2.28.1
  • google-auth from 2.40.0 to 2.43.0
  • google-cloud-core from 2.4.3 to 2.5.0
  • google-cloud-storage from 3.1.0 to 3.5.0
  • h11 from 0.14.0 to 0.16.0
  • httpcore from 1.0.2 to 1.0.9
  • httpx from 0.27.0 to 0.28.1
  • isodate from 0.6.1 to 0.7.2
  • Jinja2 from 3.1.5 to 3.1.6
  • jupyter-events from 0.10.0 to 0.12.0
  • jupyter-lsp from 2.2.0 to 2.2.5
  • jupyter_server from 2.14.1 to 2.15.0
  • jupyter_server_terminals from 0.4.4 to 0.5.3
  • mistune from 2.0.4 to 3.1.2
  • mlflow-skinny from 3.0.1 to 3.5.1
  • mmh3 from 5.1.0 to 5.2.0
  • msal from 1.32.3 to 1.34.0
  • nbclient from 0.8.0 to 0.10.2
  • nbconvert from 7.16.4 to 7.16.6
  • notebook_shim from 0.2.3 to 0.2.4
  • opentelemetry-api from 1.32.1 to 1.38.0
  • opentelemetry-sdk from 1.32.1 to 1.38.0
  • opentelemetry-semantic-conventions from 0.53b1 to 0.59b0
  • platformdirs from 3.10.0 to 4.3.7
  • prometheus_client from 0.21.0 to 0.21.1
  • psycopg2 from 2.9.3 to 2.9.11
  • pyarrow from 19.0.1 to 21.0.0
  • Pygments from 2.15.1 to 2.19.1
  • pyiceberg from 0.9.0 to 0.10.0
  • python-lsp-server from 1.12.0 to 1.12.2
  • rope from 1.12.0 to 1.13.0
  • s3transfer from 0.11.3 to 0.14.0
  • scipy from 1.15.1 to 1.15.3
  • setuptools from 74.0.0 to 78.1.1
  • six from 1.16.0 to 1.17.0
  • stack-data from 0.2.0 to 0.6.3
  • starlette from 0.46.2 to 0.49.3
  • tornado from 6.4.2 to 6.5.1
  • types-python-dateutil from 2.9.0.20241206 to 2.9.0.20251008
  • uvicorn from 0.34.2 to 0.38.0
  • webcolors from 24.11.1 to 25.10.0

Upgraded R libraries:

  • arrow from 19.0.1 to 22.0.0
  • base from 4.4.2 to 4.5.1
  • bigD from 0.3.0 to 0.3.1
  • broom from 1.0.7 to 1.0.10
  • clock from 0.7.2 to 0.7.3
  • commonmark from 1.9.5 to 2.0.0
  • compiler from 4.4.2 to 4.5.1
  • credentials from 2.0.2 to 2.0.3
  • curl from 6.4.0 to 7.0.0
  • data.table from 1.17.0 to 1.17.8
  • datasets from 4.4.2 to 4.5.1
  • dbplyr from 2.5.0 to 2.5.1
  • devtools from 2.4.5 to 2.4.6
  • diffobj from 0.3.5 to 0.3.6
  • digest from 0.6.37 to 0.6.39
  • downlit from 0.4.4 to 0.4.5
  • dtplyr from 1.3.1 to 1.3.2
  • evaluate from 1.0.3 to 1.0.5
  • fansi from 1.0.6 to 1.0.7
  • forcats from 1.0.0 to 1.0.1
  • fs from 1.6.5 to 1.6.6
  • future from 1.34.0 to 1.68.0
  • future.apply from 1.11.3 to 1.20.0
  • gargle from 1.5.2 to 1.6.0
  • gert from 2.1.4 to 2.2.0
  • ggplot2 from 3.5.1 to 4.0.1
  • gh from 1.4.1 to 1.5.0
  • git2r from 0.35.0 to 0.36.2
  • glmnet from 4.1-8 to 4.1-10
  • googledrive from 2.1.1 to 2.1.2
  • googlesheets4 from 1.1.1 to 1.1.2
  • graphics from 4.4.2 to 4.5.1
  • grDevices from 4.4.2 to 4.5.1
  • grid from 4.4.2 to 4.5.1
  • gt from 0.11.1 to 1.1.0
  • hardhat from 1.4.1 to 1.4.2
  • haven from 2.5.4 to 2.5.5
  • hms from 1.1.3 to 1.1.4
  • httpuv from 1.6.15 to 1.6.16
  • httr2 from 1.1.1 to 1.2.1
  • jsonlite from 1.9.1 to 2.0.0
  • later from 1.4.1 to 1.4.4
  • lava from 1.8.1 to 1.8.2
  • listenv from 0.9.1 to 0.10.0
  • magrittr from 2.0.3 to 2.0.4
  • markdown from 1.13 to 2.0
  • methods from 4.4.2 to 4.5.1
  • miniUI from 0.1.1.1 to 0.1.2
  • mlflow from 2.20.4 to 3.6.0
  • openssl from 2.3.3 to 2.3.4
  • parallel from 4.4.2 to 4.5.1
  • parallelly from 1.42.0 to 1.45.1
  • pillar from 1.11.0 to 1.11.1
  • pkgbuild from 1.4.6 to 1.4.8
  • pkgdown from 2.1.1 to 2.2.0
  • pkgload from 1.4.0 to 1.4.1
  • pROC from 1.18.5 to 1.19.0.1
  • prodlim from 2024.06.25 to 2025.04.28
  • progressr from 0.15.1 to 0.18.0
  • promises from 1.3.2 to 1.5.0
  • ps from 1.9.0 to 1.9.1
  • purrr from 1.0.4 to 1.2.0
  • ragg from 1.3.3 to 1.5.0
  • Rcpp from 1.0.14 to 1.1.0
  • readr from 2.1.5 to 2.1.6
  • recipes from 1.2.0 to 1.3.1
  • reshape2 from 1.4.4 to 1.4.5
  • rmarkdown from 2.29 to 2.30
  • roxygen2 from 7.3.2 to 7.3.3
  • rprojroot from 2.0.4 to 2.1.1
  • RSQLite from 2.3.9 to 2.4.4
  • rversions from 2.1.2 to 3.0.0
  • rvest from 1.0.4 to 1.0.5
  • sass from 0.4.9 to 0.4.10
  • scales from 1.3.0 to 1.4.0
  • shiny from 1.10.0 to 1.11.1
  • sparklyr from 1.9.1 to 1.9.3
  • SparkR from 4.0.0 to 4.1.0
  • sparsevctrs from 0.3.1 to 0.3.4
  • splines from 4.4.2 to 4.5.1
  • stats from 4.4.2 to 4.5.1
  • stats4 from 4.4.2 to 4.5.1
  • stringr from 1.5.1 to 1.6.0
  • systemfonts from 1.2.1 to 1.3.1
  • tcltk from 4.4.2 to 4.5.1
  • testthat from 3.2.3 to 3.3.0
  • textshaping from 1.0.0 to 1.0.4
  • timeDate from 4041.110 to 4051.111
  • tinytex from 0.56 to 0.58
  • tools from 4.4.2 to 4.5.1
  • usethis from 3.1.0 to 3.2.1
  • utils from 4.4.2 to 4.5.1
  • V8 from 6.0.2 to 8.0.1
  • vroom from 1.6.5 to 1.6.6
  • waldo from 0.6.1 to 0.6.2
  • xfun from 0.51 to 0.54
  • xml2 from 1.3.8 to 1.5.0
  • zeallot from 0.1.0 to 0.2.0
  • zip from 2.3.2 to 2.3.3

Upgraded Java libraries:

  • com.amazonaws.amazon-kinesis-client from 1.12.0 to 1.15.3
  • com.amazonaws.aws-java-sdk-autoscaling from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudformation from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudfront from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudhsm from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudsearch from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudtrail from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudwatch from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cloudwatchmetrics from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-codedeploy from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cognitoidentity from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-cognitosync from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-config from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-core from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-datapipeline from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-directconnect from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-directory from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-dynamodb from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-ec2 from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-ecs from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-efs from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-elasticache from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-elasticbeanstalk from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-elasticloadbalancing from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-elastictranscoder from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-emr from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-glacier from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-glue from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-iam from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-importexport from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-kinesis from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-kms from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-lambda from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-logs from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-machinelearning from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-opsworks from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-rds from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-redshift from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-route53 from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-s3 from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-ses from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-simpledb from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-simpleworkflow from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-sns from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-sqs from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-ssm from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-storagegateway from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-sts from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-support from 1.12.638 to 1.12.681
  • com.amazonaws.aws-java-sdk-workspaces from 1.12.638 to 1.12.681
  • com.amazonaws.jmespath-java from 1.12.638 to 1.12.681
  • com.databricks.databricks-sdk-java from 0.27.0 to 0.53.0
  • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.18.2 to 2.18.3
  • com.github.luben.zstd-jni from 1.5.6-10 to 1.5.7-6
  • com.google.flatbuffers.flatbuffers-java from 24.3.25 to 25.2.10
  • com.google.guava.failureaccess from 1.0.2 to 1.0.3
  • com.google.guava.guava from 33.4.0-jre to 33.4.8-jre
  • com.microsoft.sqlserver.mssql-jdbc from 11.2.3.jre8 to 12.8.0.jre8
  • commons-cli.commons-cli from 1.9.0 to 1.10.0
  • commons-codec.commons-codec from 1.17.2 to 1.19.0
  • commons-fileupload.commons-fileupload from 1.5 to 1.6.0
  • commons-io.commons-io from 2.18.0 to 2.21.0
  • dev.ludovic.netlib.arpack from 3.0.3 to 3.0.4
  • dev.ludovic.netlib.blas from 3.0.3 to 3.0.4
  • dev.ludovic.netlib.lapack from 3.0.3 to 3.0.4
  • io.delta.delta-sharing-client_2.13 from 1.3.5 to 1.3.6
  • io.dropwizard.metrics.metrics-annotation from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-core from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-graphite from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-healthchecks from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-jmx from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-json from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-jvm from 4.2.30 to 4.2.37
  • io.dropwizard.metrics.metrics-servlets from 4.2.30 to 4.2.37
  • io.netty.netty-all from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-buffer from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-codec from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-codec-http from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-codec-http2 from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-codec-socks from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-common from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-handler from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-handler-proxy from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-resolver from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-tcnative-boringssl-static from 2.0.70.Final-db-r0-windows-x86_64 to 2.0.74.Final-db-r0-windows-x86_64
  • io.netty.netty-tcnative-classes from 2.0.70.Final to 2.0.74.Final
  • io.netty.netty-transport from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-transport-classes-epoll from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-transport-classes-kqueue from 4.1.118.Final to 4.2.7.Final
  • io.netty.netty-transport-native-epoll from 4.1.118.Final-linux-x86_64 to 4.2.7.Final-linux-x86_64
  • io.netty.netty-transport-native-kqueue from 4.1.118.Final-osx-x86_64 to 4.2.7.Final-osx-x86_64
  • io.netty.netty-transport-native-unix-common from 4.1.118.Final to 4.2.7.Final
  • joda-time.joda-time from 2.13.0 to 2.14.0
  • org.apache.arrow.arrow-format from 18.2.0 to 18.3.0
  • org.apache.arrow.arrow-memory-core from 18.2.0 to 18.3.0
  • org.apache.arrow.arrow-memory-netty from 18.2.0 to 18.3.0
  • org.apache.arrow.arrow-memory-netty-buffer-patch from 18.2.0 to 18.3.0
  • org.apache.arrow.arrow-vector from 18.2.0 to 18.3.0
  • org.apache.avro.avro from 1.12.0 to 1.12.1
  • org.apache.avro.avro-ipc from 1.12.0 to 1.12.1
  • org.apache.avro.avro-mapred from 1.12.0 to 1.12.1
  • org.apache.commons.commons-collections4 from 4.4 to 4.5.0
  • org.apache.commons.commons-compress from 1.27.1 to 1.28.0
  • org.apache.commons.commons-lang3 from 3.17.0 to 3.19.0
  • org.apache.commons.commons-text from 1.13.0 to 1.14.0
  • org.apache.curator.curator-client from 5.7.1 to 5.9.0
  • org.apache.curator.curator-framework from 5.7.1 to 5.9.0
  • org.apache.curator.curator-recipes from 5.7.1 to 5.9.0
  • org.apache.datasketches.datasketches-java from 6.1.1 to 6.2.0
  • org.apache.hadoop.hadoop-client-runtime from 3.4.1 to 3.4.2
  • org.apache.orc.orc-core from 2.1.1-shaded-protobuf to 2.2.0-shaded-protobuf
  • org.apache.orc.orc-format from 1.1.0-shaded-protobuf to 1.1.1-shaded-protobuf
  • org.apache.orc.orc-mapreduce from 2.1.1-shaded-protobuf to 2.2.0-shaded-protobuf
  • org.apache.orc.orc-shims from 2.1.1 to 2.2.0
  • org.apache.xbean.xbean-asm9-shaded from 4.26 to 4.28
  • org.apache.zookeeper.zookeeper from 3.9.3 to 3.9.4
  • org.apache.zookeeper.zookeeper-jute from 3.9.3 to 3.9.4
  • org.eclipse.jetty.jetty-client from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-http from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-io from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-jndi from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-plus from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-proxy from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-security from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-server from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-servlet from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-servlets from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-util from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-webapp from 9.4.53.v20231009 to 10.0.26
  • org.eclipse.jetty.jetty-xml from 9.4.53.v20231009 to 10.0.26
  • org.mlflow.mlflow-spark_2.13 from 2.9.1 to 2.22.1
  • org.objenesis.objenesis from 3.3 to 3.4
  • org.scala-lang.modules.scala-xml_2.13 from 2.3.0 to 2.4.0

Apache Spark​

Databricks Runtime 18.0 (Beta) includes Apache Spark 4.0.0. This release includes all Spark fixes and improvements included in the previous version, as well as the following additional bug fixes and improvements made to Spark:

  • SPARK-54536 Shuffle FetchWaitTime missing collect create client/wait cost
  • SPARK-54534 Migrate Hive-related legacy error codes to proper error conditions
  • SPARK-54565 SparkBuildInfo should load spark-version-info.properties from its own classloader
  • SPARK-54533 Set metric ExecutorSource.METRIC_RESULT_SIZE with correct value
  • SPARK-54478 Re-enable streaming tests for connect compat test CI
  • SPARK-54552 Fix SparkConnectResultSet.getString to handle BINARY data type with UTF_8
  • SPARK-54501 Improve error handling for Hive metastore partition filter failures
  • SPARK-54550 Handle ConnectException gracefully in SparkConnectStatement.close()
  • SPARK-54020 Support spark.sql(...) Python API inside query functions for Spark Declarative Pipeline
  • SPARK-53127 Fix LIMIT ALL for unlimted recursion with CTE normalization
  • SPARK-50072 Handle ArithmeticException in interval parsing with large values
  • SPARK-54299 Fix the wrong example query in WindowGroupLimit
  • SPARK-54505 Correct the arguments order of createMetrics call in makeNegative
  • SPARK-54462 Add SupportsV1OverwriteWithSaveAsTable mixin for TableProvider
  • SPARK-54540 Couples of minor fix for Connect JDBC driver
  • SPARK-54508 Fix spark-pipelines to resolve spec file path more robustly
  • SPARK-54087 Spark Executor launch task failed should return task killed message
  • SPARK-53797 Fix FileStreamSource.takeFilesUntilMax to use zipWithIndex to avoid indices usage
  • SPARK-54418 Fix error messages and code formatting
  • SPARK-54114 Support getColumns for SparkConnectDatabaseMetaData
  • SPARK-54209 Support TIMESTAMP type in SparkConnectResultSet
  • SPARK-54208 Support TIME type in SparkConnectResultSet
  • SPARK-54528 Close URLClassLoader eagerly to avoid OOM
  • SPARK-54464 Remove duplicate output.reserve calls in assembleVariantBatch
  • SPARK-53635 Support Scala UDFs with input args of type Seq[Row]
  • SPARK-54493 Fix assertSchemaEqual for MapType
  • SPARK-52515 Test approx_top_k with flag on and off
  • SPARK-54413 Upgrade Bootstrap v4.4.1 to v4.6.2
  • SPARK-54497 Apply functools.lru_cache in converter caching
  • SPARK-54306 Annotate Variant columns with Variant logical type annotation
  • SPARK-54350 SparkGetColumnsOperation ORDINAL_POSITION should be 1-based
  • SPARK-54130 Add detailed error messages for catalog assertion failures
  • SPARK-54220 NullType/VOID/UNKNOWN Type Support in Parquet
  • SPARK-54163 Scan canonicalization for partitioning and ordering info
  • SPARK-54377 Fix COMMENT ON TABLE IS NULL to properly remove table comment
  • SPARK-52767 Optimize maxRows and maxRowsPerPartition for join and union
  • SPARK-54063 Trigger snapshot for next batch when upload lag
  • SPARK-54384 Modernize the _batched method for BatchedSerializer
  • SPARK-54378 Remove CreateXmlParser.scala from catalyst module
  • SPARK-53103 Revert "[SC-204946][ss] Throw an error if state directory is not empty when query starts"
  • SPARK-53103 Throw an error if state directory is not empty when query starts
  • SPARK-54397 Make UserDefinedType hashable
  • SPARK-54440 Give default pipeline spec file more idiomatic name, spark-pipeline.yml
  • SPARK-54324 Add test for client-user-context-extensions
  • SPARK-54456 Import worker module after fork to avoid deadlock
  • SPARK-54427 Allow ColumnarRow to call copy with variant types
  • SPARK-54136 Extract plan merging logic from MergeScalarSubqueries to PlanMerger
  • SPARK-54389 Fix RocksDB State Store Invalid Stamp error when task is marked as failed during initialization
  • SPARK-54346 Introduce state repartition API and repartition runner
  • SPARK-53809 Add canonicalization for DataSourceV2ScanRelation
  • SPARK-54280 Require pipeline checkpoint storage dir to be absolute path
  • SPARK-54206 Support BINARY type data in SparkConnectResultSet
  • SPARK-54319 BHJ LeftAnti update numOutputRows wrong when codegen is disabled
  • SPARK-54355 Make spark.connect.session.planCompression.defaultAlgorithm to support NONE
  • SPARK-54341 Remember TimeTravelSpec for tables loaded via TableProvider
  • SPARK-54280 Revert "[SC-212148][sdp] Require pipeline checkpoint storage dir to be absolute path"
  • SPARK-54354 Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation
  • SPARK-54439 KeyGroupedPartitioning and join key size mismatch
  • SPARK-54280 Require pipeline checkpoint storage dir to be absolute path
  • SPARK-54395 RemoteBlockPushResolver class repeatedly initializes ObjectMapper
  • SPARK-54207 Supports Date type data in SparkConnectResultSet
  • SPARK-54182 Optimize non-arrow conversion of df.toPandas
  • SPARK-54312 Avoid repeatedly scheduling tasks for SendHeartbeat/WorkDirClean in standalone worker
  • SPARK-54394 Move isJavaVersionAtMost17 and isJavaVersionAtLeast21 from core to common/utils
  • SPARK-53927 Upgrade kinesis client
  • SPARK-54358 Checkpoint dirs collide when streaming tables in different schemas have same name
  • SPARK-54310 Revert "[SC-212003][sql] Add numSourceRows metric for MergeIntoExec"
  • SPARK-54310 Add numSourceRows metric for MergeIntoExec
  • SPARK-53322 Select a KeyGroupedShuffleSpec only when join key positions can be fully pushed down
  • SPARK-54270 SparkConnectResultSet get* methods should call checkOpen and check index boundary
  • SPARK-53849 Update netty and netty tc native
  • SPARK-54205 Supports Decimal type data in SparkConnectResultSet
  • SPARK-54417 Fix error message for scalar subquery in IDENTIFIER clause
  • SPARK-54113 Support getTables for SparkConnectDatabaseMetaData
  • SPARK-54303 Canonicalize error condition MISSING_CATALOG_ABILITY
  • SPARK-54153 Support profiling iterator based Python UDFs
  • SPARK-54349 Refactor code a bit to simplify faulthandler integration extension
  • SPARK-54317 Unify Arrow conversion logic for Classic and Connect toPandas
  • SPARK-54339 Fix AttributeMap non-determinism
  • SPARK-54112 Support getSchemas for SparkConnectDatabaseMetaData
  • SPARK-54180 Override the toString of BinaryFileFormat
  • SPARK-54213 Remove Python 3.9 from Spark Connect
  • SPARK-54215 Add the SessionStateHelper trait to FilePartition
  • SPARK-54115 Escalate display ordering priority of connect server operation threads in thread dump page
  • SPARK-54193 Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent
  • SPARK-54149 Enable tail-recursion wherever possible
  • SPARK-54185 Deprecated spark.shuffle.server.chunkFetchHandlerThreadsPercent
  • SPARK-54056 resolve substitution for SQLConf settings in Catalogs
  • SPARK-54147 Set OMP_NUM_THREADS to spark.task.cpus by default in BaseScriptTransformationExec
  • SPARK-54229 Make PySparkLogger in UDFs store one log entry per log function call
  • SPARK-53337 XSS: Ensure the application name in historypage get escaped
  • SPARK-54229 Revert "[SC-211321][python] Make PySparkLogger in UDFs store one log entry per log function call"
  • SPARK-54373 Enlarge the SVG viewBox attribute of Job DAG initialization
  • SPARK-54323 Change the way to access logs to TVF instead of system view
  • SPARK-54229 Make PySparkLogger in UDFs store one log entry per log function call
  • SPARK-53978 Support logging in driver-side workers
  • SPARK-54146 Clean up the usage of deprecated Jackson API
  • SPARK-54383 Add precomputed schema variant for InternalRowComparableWrapper util
  • SPARK-54030 Add user-friendly assertion message for view metadata corruption
  • SPARK-54144 Short Circuit Eval Type Inferences
  • SPARK-54030 Revert "[SC-210301][sql] Add user-friendly assertion message for view metadata corruption"
  • SPARK-54028 Use empty schema when altering a view which is not Hive compatible
  • SPARK-54030 Add user-friendly assertion message for view metadata corruption
  • SPARK-54085 Fix initialize to add CREATE option additionally in DriverRunner
  • SPARK-53482 MERGE INTO support for when source has less nested field than target
  • SPARK-53905 Refactor RelationResolution to enable code reuse
  • SPARK-53732 Remember TimeTravelSpec in DataSourceV2Relation
  • SPARK-54014 Support max rows for SparkConnectStatement
  • SPARK-50906 Fix Avro nullability check for reordered struct fields
  • SPARK-54396 Optimize Py4J calls in Dataframe.toArrow
  • SPARK-54344 Kill the worker if flush fails in daemon.py
  • SPARK-53977 Support logging in UDTFs
  • SPARK-52515 Reapply "[SC-199815][sql] Add approx_top_k function"
  • SPARK-54340 Add the capability to use viztracer on pyspark daemon/workers
  • SPARK-54379 Move lambda binding to separate LambdaBinder object
  • SPARK-54029 Add detailed error message for table metadata corruption to ease debugging
  • SPARK-54002 Support integrating BeeLine with Connect JDBC driver
  • SPARK-54336 Fix BloomFilterMightContain input type check with ScalarSubqueryReference
  • SPARK-53406 Avoid unnecessary shuffle join in direct passthrough shuffle id
  • SPARK-54347 Optimize Py4J calls in classic dataframe
  • SPARK-54062 MergeScalarSubqueries code cleanup
  • SPARK-54054 Support row position for SparkConnectResultSet
  • SPARK-54330 Optimize Py4J calls in spark.createDataFrame
  • SPARK-54332 No need to attach PlanId in grouping column names in rollup/cube/groupingSets
  • SPARK-53976 Support logging in Pandas/Arrow UDFs
  • SPARK-54123 Add timezone to make the timestamp an absolute time
  • SPARK-54356 Fix EndToEndAPISuite caused by missing storage root schema
  • SPARK-54292 Support aggregate functions and GROUP BY in |> SELECT pipe operators (#180106) (#180368)
  • SPARK-54376 Mark most pipeline configuration options as internal
  • SPARK-53975 Adds basic Python worker logging support
  • SPARK-54361 Fix Spark version to intended value of 4.2.0 for spark.sql.parser.singleCharacterPipeOperator.enabled
  • SPARK-51518 Support | as an alternative to |> for the SQL pipe operator token
  • SPARK-53535 Fix missing structs always being assumed as nulls
  • SPARK-54294 Normalize Connect server printed IP address
  • SPARK-52439 Support check constraint with null value
  • SPARK-54352 Introduce SQLConf.canonicalize to centralize normalization of strings
  • SPARK-54183 Revert Revert "[SC-211824][python][CONNECT] Avoid one intermediate temp data frame during spark connect toPandas()"
  • SPARK-53573 IDENTIFIER everywhere
  • SPARK-53573 Allow coalescing string literals everywhere
  • SPARK-54240 Translate get array item catalyst expression to connector expression
  • SPARK-54334 Move the validation of subquery expressions under lambda and higher order functions to SubqueryExpressionInLambdaOrHigherOrderFunctionValidator
  • SPARK-54183 Revert "[SC-211824][python][CONNECT] Avoid one intermediate temp data frame during spark connect toPandas()"
  • SPARK-54183 Avoid one intermediate temp data frame during spark connect toPandas()
  • SPARK-54264 DeDup Operator can use RocksDB's keyExists()
  • SPARK-54269 Upgrade cloudpickle to 3.1.2 for Python 3.14
  • SPARK-54300 Optimize Py4J calls in df.toPandas
  • SPARK-54307 Throw an error if streaming query is restarted with stateful op but there is empty state dir
  • SPARK-54117 Throw better error to indicate that TWS is only supported with RocksDB state store provider
  • SPARK-53917 Support large local relations - follow-ups
  • SPARK-54275 Clean up unused code from pipeline module
  • SPARK-54287 Add Python 3.14 support in pyspark-client and pyspark-connect
  • SPARK-53614 Add Iterator[pandas.DataFrame] support to applyInPandas
  • SPARK-54191 Add once to Defineflow Proto
  • SPARK-54234 Not need to attach PlanId in grouping column names in df.groupBy
  • SPARK-54231 Fill gaps in SDP docs
  • SPARK-54199 Add DataFrame API support for new KLL quantiles sketch functions (#178526)
  • SPARK-52463 Reapply "[SC-211221][sdp] Add support for cluster_by in Python Pipelines APIs"
  • SPARK-52463 Revert "[SC-211221][sdp] Add support for cluster_by in Python Pipelines APIs"
  • SPARK-53786 Default value with special column name should not conflict with real column
  • SPARK-54200 Update KafkaConsumerPoolRealTimeModeSuite to use groupIdPrefix for concurrent runs safety
  • SPARK-52463 Add support for cluster_by in Python Pipelines APIs
  • SPARK-52509 Cleanup individual shuffles from fallback storage on RemoveShuffle event
  • SPARK-54187 Reapply "[SC-211150][python][CONNECT] Get all configs in a batch in toPandas"
  • SPARK-53942 Support changing stateless shuffle partitions upon restart of streaming query
  • SPARK-52515 Revert "[SC-199815][sql] Add approx_top_k function"
  • SPARK-52515 Add approx_top_k function
  • SPARK-54078 New test for StateStoreSuite SPARK-40492: maintenance before unload and remove infra from old test
  • SPARK-54178 Improve error for ResolveSQLOnFile
  • SPARK-53455 Add CloneSession RPC
  • SPARK-54178 Revert "[SC-211111][sql] Improve error for ResolveSQLOnFile"
  • SPARK-53489 Remove using v2ColumnsToStructType in ApplyDefaultCollationToStringType
  • SPARK-54178 Improve error for ResolveSQLOnFile
  • SPARK-54187 Revert "[SC-211150][python][CONNECT] Get all configs in a batch in toPandas"
  • SPARK-53942 Revert "[SC-209873][ss] Support changing stateless shuffle partitions upon restart of streaming query"
  • SPARK-54187 Get all configs in a batch in toPandas
  • SPARK-54145 Fix column check of nested type in numeric aggregation
  • SPARK-53942 Support changing stateless shuffle partitions upon restart of streaming query
  • SPARK-53991 Add SQL support for KLL quantiles functions based on DataSketches (#178089) (#178234)
  • SPARK-53128 Include unmanaged memory bytes in the usage log before execution memory OOM
  • SPARK-53731 Update the type hints of iterator APIs
  • SPARK-53967 Avoid intermediate pandas dataframe creation in df.toPandas
  • SPARK-53455 Revert "[SC-208758][connect] Add CloneSession RPC"
  • SPARK-54108 Revise execute* methods of SparkConnectStatement
  • SPARK-54052 Add a bridge object to workaround Py4J limitation
  • SPARK-54128 Convert IllegalArgumentException to SparkException with proper error classes in Spark Connect server
  • SPARK-53880 Fix DSv2 in PushVariantIntoScan by adding SupportsPushDownVariants
  • SPARK-54111 Support getCatalogs for SparkConnectDatabaseMetaData
  • SPARK-53455 Add CloneSession RPC
  • SPARK-54118 Improve the put/merge operation in ListState when there are multiple values
  • SPARK-54038 Support getSQLKeywords for SparkConnectDatabaseMetaData
  • SPARK-54013 Implement SparkConnectDatabaseMetaData simple methods
  • SPARK-53934 Initial implement Connect JDBC driver
  • SPARK-53959 Throw a client-side error when creating a dataframe from a pandas dataframe with an index but no data
  • SPARK-53573 Revert "[SC-210255][sql] Allow coalescing string literals everywhere"
  • SPARK-54094 Extract common methods to KafkaOffsetReaderBase
  • SPARK-53573 Allow coalescing string literals everywhere
  • SPARK-54039 Add TaskContext information to KafkaDataConsumer release() logs for better debugging
  • SPARK-54031 Add new golden file tests for analysis edge-cases
  • SPARK-54067 Improve SparkSubmit to invoke exitFn with the root cause instead of SparkUserAppException
  • SPARK-54047 Use a difference error when kill-on-idle-timeout
  • SPARK-54078 Deflake StateStoreSuite SPARK-40492: maintenance before unload
  • SPARK-54061 Wrap IllegalArgumentException with proper error code for invalid datetime patterns
  • SPARK-54031 Revert "[SC-210084][sql] Add new golden file tests for analysis edge-cases"
  • SPARK-54075 Make ResolvedCollation evaluable
  • SPARK-54001 Optimize memory usage in session cloning with ref-counted cached local relations
  • SPARK-54031 Add new golden file tests for analysis edge-cases
  • SPARK-53923 Rename spark.executor.(log -> logs).redirectConsoleOutputs
  • SPARK-54007 Use Java Set.of instead of Collections.emptySet
  • SPARK-53755 Add log support in BlockManager
  • SPARK-54041 Refactor ParameterizedQuery arguments validation
  • SPARK-53696 Default to bytes for BinaryType in PySpark
  • SPARK-53921 Introduce GeometryType and GeographyType to PySpark API
  • SPARK-53788 Move VersionUtils to common module
  • SPARK-53999 Native KQueue Transport support on BSD/MacOS
  • SPARK-54021 Implement Geography and Geometry accessors across Catalyst
  • SPARK-53921 Revert "[SC-209482][geo][PYTHON] Introduce GeometryType and GeographyType to PySpark API"
  • SPARK-53920 Introduce GeometryType and GeographyType to Java API
  • SPARK-53610 Limit Arrow batch sizes in CoGrouped applyInPandas and applyInArrow
  • SPARK-53659 Infer Variant shredding schema when writing to Parquet
  • SPARK-53922 Introduce physical Geometry and Geography types
  • SPARK-54059 Reduce default page size by LONG_ARRAY_OFFSET if ZGC or ShenandoahGC and ON_HEAP are used
  • SPARK-53921 Introduce GeometryType and GeographyType to PySpark API
  • SPARK-54048 Update dev/requirements.txt to install torch(vision) in Python 3.14
  • SPARK-53917 Support large local relations
  • SPARK-53760 Introduce GeometryType and GeographyType
  • SPARK-53530 Clean up the useless code related to TransformWithStateInPySparkStateServer
  • SPARK-53636 Fix thread-safety issue in SortShuffleManager.unregisterShuffle
  • SPARK-52762 Add PipelineAnalysisContext message to support pipeline analysis during Spark Connect query execution
  • SPARK-53631 Optimize memory and perf on SHS bootstrap
  • SPARK-53857 Enable messageTemplate propagation to SparkThrowable
  • SPARK-53891 Model DSV2 Commit Write Summary API
  • SPARK-53966 Add utility functions to detect JVM GCs
  • SPARK-53149 Fix testing whether BeeLine process run in background
  • SPARK-53738 Fix planned write when query output contains foldable orderings
  • SPARK-53949 Use Utils.getRootCause instead of Throwables.getRootCause
  • SPARK-53696 Revert "[SC-209330][python][CONNECT][sql] Default to bytes for BinaryType in PySpark"
  • SPARK-53804 Support TIME radix sort
  • SPARK-54004 Fix uncaching table by name without cascading
  • SPARK-53261 Use Java String.join|StringJoiner instead of Guava Joiner
  • SPARK-53319 Support the time type by try_make_timestamp_ltz()
  • SPARK-53280 Use Java instanceof instead of Throwables.throwIf* methods
  • SPARK-53696 Default to bytes for BinaryType in PySpark
  • SPARK-53258 Use JavaUtils's check(Argument|State)
  • SPARK-53773 Recover alphabetic ordering of rules in RuleIdCollection
  • SPARK-53256 Promote check(Argument|State) to JavaUtils
  • SPARK-54008 Skip QO for DESCRIBE QUERY
  • SPARK-53573 Use Pre-processor for generalized parameter marker handling
  • SPARK-53980 Add SparkConf.getAllWithPrefix(String, String => K) API
  • SPARK-54009 Support spark.io.mode.default
  • SPARK-51903 Validate data on adding a CHECK constraint
  • SPARK-53573 Revert "[SC-209126][sql] Use Pre-processor for generalized parameter marker handling"
  • SPARK-53573 Use Pre-processor for generalized parameter marker handling
  • SPARK-53573 Revert "[SC-209126][sql] Use Pre-processor for generalized parameter marker handling"
  • SPARK-53956 Support TIME in the try_make_timestamp function in PySpark
  • SPARK-53930 Support TIME in the make_timestamp function in PySpark
  • SPARK-53573 Use Pre-processor for generalized parameter marker handling
  • SPARK-53564 Avoid DAGScheduler exits due to blockManager RPC timeout in DAGSchedulerEventProcessLoop
  • SPARK-53879 Upgrade Ammonite to 3.0.3
  • SPARK-53938 Fix decimal rescaling in LocalDataToArrowConversion
  • SPARK-53845 SDP Sinks
  • SPARK-53908 Fix observations on Spark Connect with plan cache
  • SPARK-53841 Implement transform() in Column API
  • SPARK-53929 Support TIME in the make_timestamp and try_make_timestamp functions in Scala
  • SPARK-53902 Add tree node pattern bits for supported expressions in ParameterizedQuery argument list
  • SPARK-53064 Rewrite MDC LogKey in Java
  • SPARK-53762 Add date and time conversions simplifier rule to optimizer
  • SPARK-53916 Deduplicate the variables in PythonArrowInput
  • SPARK-53690 Fix exponential formatting of avgOffsetsBehindLatest and estimatedTotalBytesBehindLatest in Kafka sources object in progress json
  • SPARK-53789 Canonicalize error condition CANNOT_MODIFY_STATIC_CONFIG
  • SPARK-53720 Simplify extracting Table from DataSourceV2Relation (#52460)
  • SPARK-45530 Use java.lang.ref.Cleaner instead of finalize for NioBufferedFileInputStream
  • SPARK-53789 Revert "[SC-208902][sql][CONNECT] Canonicalize error condition CANNOT_MODIFY_STATIC_CONFIG"
  • SPARK-53789 Canonicalize error condition CANNOT_MODIFY_STATIC_CONFIG
  • SPARK-53111 Implement the time_diff function in PySpark
  • SPARK-53896 Enable spark.io.compression.lzf.parallel.enabled by default
  • SPARK-53856 Remove denylist alternative config names
  • SPARK-53611 Limit Arrow batch sizes in window agg UDFs
  • SPARK-53575 Retry entire consumer stages when checksum mismatch detected for a retried shuffle map task
  • SPARK-53867 Limit Arrow batch sizes in SQL_GROUPED_AGG_ARROW_UDF
  • SPARK-53877 Introduce BITMAP_AND_AGG function
  • SPARK-51426 Fix 'Setting metadata to empty dict does not work'
  • SPARK-53868 Use array length check instead of direct reference check in V2ExpressionBuilder
  • SPARK-53609 Limit Arrow batch sizes in SQL_GROUPED_AGG_PANDAS_UDF
  • SPARK-53892 Use DescribeTopicsResult.allTopicNames instead of the deprecated all API
  • SPARK-53878 Fix race condition issue related to ObservedMetrics
  • SPARK-53796 Add extension field to a few pipeline protos to support forward compatibility
  • SPARK-53850 Define proto for Sinks and Rename DefineDataset to DefineOutput
  • SPARK-53812 Refactor DefineDataset and DefineFlow protos to group related properties and future-proof
  • SPARK-53870 Fix partial read bug for large proto messages in TransformWithStateInPySparkStateServer
  • SPARK-53751 Explicit Versioned Checkpoint Location
  • SPARK-52407 Add support for Theta Sketch (#171135)
  • SPARK-53779 Implement transform() in Column API
  • SPARK-49547 Add iterator of RecordBatch API to applyInArrow
  • SPARK-53802 Support string values for user-specified schema in SDP tables
  • SPARK-53865 Extract common logic from ResolveGenerate rule
  • SPARK-53113 Support the time type by try_make_timestamp()
  • SPARK-53868 Only use signature with Expression[] of visitAggregateFunction in V2ExpressionSQBuilder
  • SPARK-53792 Fix rocksdbPinnedBlocksMemoryUsage when bounded memory …
  • SPARK-53248 Support checkedCast in JavaUtils
  • SPARK-52640 Propagate Python Source Code Location
  • SPARK-52924 Support ZSTD_strategy for compression
  • SPARK-53562 Reapply "[SC-207233][python] Limit Arrow batch sizes in applyInArrow and applyInPandas"
  • SPARK-51272 Aborting instead of continuing partially completed indeterminate result stage at ResubmitFailedStages
  • SPARK-53795 Remove unused parameters in LiteralValueProtoConverter
  • SPARK-53808 Allow to pass optional JVM args to spark-connect-scala-client
  • SPARK-52614 Support RowEncoder inside Product Encoder
  • SPARK-53833 Update dev/requirements.txt to skip torch/torchvision in Python 3.14
  • SPARK-53715 Refactor getWritePrivileges for MergeIntoTable
  • SPARK-53516 Fix spark.api.mode arg process in SparkPipelines
  • SPARK-53507 Don't use case class for BreakingChangeInfo
  • SPARK-53645 Implement skipna parameter for ps.DataFrame any()
  • SPARK-53717 Revise MapType.valueContainsNull parameter comment more correctly
  • SPARK-53700 Remove redundancy in DataSourceV2RelationBase.simpleString
  • SPARK-53667 Fix EXPLAIN for CALL with IDENTIFIER
  • SPARK-53562 Revert "[SC-207233][python] Limit Arrow batch sizes in applyInArrow and applyInPandas"
  • SPARK-51169 Add Python 3.14 support in Spark Classic
  • SPARK-53562 Limit Arrow batch sizes in applyInArrow and applyInPandas
  • SPARK-53806 Allow empty input on AES decrypt to have error class
  • SPARK-51756 Computes RowBasedChecksum in ShuffleWriters
  • SPARK-52807 Proto changes to support analysis inside Declarative Pipelines query functions
  • SPARK-53728 Print PipelineEvent Message with Error In Test
  • SPARK-53207 Send Pipeline Event to Client Asynchronously
  • SPARK-53829 Support datetime.time in column operators
  • SPARK-53638 Limit the byte size of arrow batch for TWS to avoid OOM
  • SPARK-53593 Fix: Use unquoted for response fields
  • SPARK-53734 Prefer table column over LCA when resolving array index
  • SPARK-53593 Add response field for DefineDataset and DefineFlow RPC
  • SPARK-53734 Revert "[SC-207697][sql] Prefer table column over LCA when resolving array index"
  • SPARK-53621 Adding Support for Executing CONTINUE HANDLER
  • SPARK-53734 Prefer table column over LCA when resolving array index
  • SPARK-53507 Add breaking change info to errors
  • SPARK-53707 Improve attribute metadata handling.
  • SPARK-53629 Implement type widening for MERGE INTO WITH SCHEMA EVOLUTION
  • SPARK-53719 Enhance type checking in _to_col function
  • SPARK-53735 Hide server-side JVM stack traces by default in spark-pipelines output
  • SPARK-53651 Add support for persistent views in pipelines
  • SPARK-53678 Fix NPE when subclass of ColumnVector is created with null DataType
  • SPARK-46679 Fix for SparkUnsupportedOperationException Not found an encoder of the type T, when using Parameterized class
  • SPARK-53444 Rework execute immediate rework execute immediate
  • SPARK-53578 Simplify data type handling in LiteralValueProtoConverter
  • SPARK-53438 Use CatalystConverter in LiteralExpressionProtoConverter
  • SPARK-53444 Revert "[SC-206535][sql][CORE] Rework execute immediate rework execute immediate"
  • SPARK-53444 Rework execute immediate rework execute immediate
  • SPARK-53112 Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark
  • SPARK-53492 Reject second ExecutePlan with an operation id that has already completed
  • SPARK-52772 Forward fixing gold files
  • SPARK-53591 Simplify Pipeline Spec Pattern Glob Matching
  • SPARK-53553 Fix handling of null values in LiteralValueProtoConverter
  • SPARK-52772 Inconsistent table attribute handling during updates
  • SPARK-53544 Support complex types on observations
  • SPARK-53357 Update pandas to 2.3.2
  • SPARK-53402 Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
  • SPARK-53372 SDP End to End Testing Suite
  • SPARK-53402 Revert "[SC-206163][connect] Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala"
  • SPARK-53402 Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
  • SPARK-53233 Make the code related to streaming uses the correct package name
  • SPARK-53233 Revert "[SC-203716][sql][SS][mllib][CONNECT] Make the code related to streaming uses the correct package name"
  • SPARK-53233 Make the code related to streaming uses the correct package name
  • SPARK-53561 Catch Interruption Exception in TransformWithStateInPySparkStateServer during outputStream.flush to avoid the worker crash
  • SPARK-53486 Avoid setting weights_only=False in torch model load
  • SPARK-53391 Remove unused PrimitiveKeyOpenHashMap
  • SPARK-52431 Finishing touches on Declarative Pipelines runner
  • SPARK-52980 Support Arrow Python UDTFs
  • SPARK-53453 Unblock 'torch<2.6.0'
  • SPARK-52582 Remerge "[SC-199314][sc-204581][SQL] Improve the memory usage of XML parser"
  • SPARK-52582 Revert "[SC-199314][sc-204581][SQL] Improve the memory usage of XML parser"
  • SPARK-52582 Improve the memory usage of XML parser
  • SPARK-51906 Dsv2 expressions in alter table add columns
  • SPARK-53150 Improve list(File|Path)s to handle non-existent, non-directory, symlink input
  • SPARK-53091 Ban org.apache.commons.lang3.Strings
  • SPARK-52844 Update numpy to 1.22
  • SPARK-52912 Improve SparkStringUtils to support is(Not)?(Blank|Empty)
  • SPARK-52877 Improve Python UDF Arrow Serializer Performance
  • SPARK-52877 Revert "[SC-201914][python] Improve Python UDF Arrow Serializer Performance"
  • SPARK-52238 Python client for Declarative Pipelines
  • SPARK-52877 Improve Python UDF Arrow Serializer Performance
  • SPARK-53287 Add ANSI Migration Guide
  • SPARK-52110 Implement SQL syntax support for pipelines
  • SPARK-52897 Update pandas to 2.3.1
  • SPARK-52451 Revert "[SC-202894][connect][SQL] Make WriteOperation in SparkConnectPlanner side effect free"
  • SPARK-52976 Fix Python UDF not accepting collated string as input param/return type
  • SPARK-52904 Enable convertToArrowArraySafely by default
  • SPARK-52821 add int->DecimalType pyspark udf return type coercion
  • SPARK-52971 Limit idle Python worker queue size
  • SPARK-53013 Fix Arrow-optimized Python UDTF returning no rows on lateral join
  • SPARK-51834 Sync OSS: support end-to-end table constraint alter add/drop constraint
  • SPARK-52675 Interrupt hanging ML handlers in tests
  • SPARK-52959 Support UDT in Arrow-optimized Python UDTF
  • SPARK-52961 Fix Arrow-optimized Python UDTF with 0-arg eval on lateral join
  • SPARK-52949 Avoid roundtrip between RecordBatch and Table in Arrow-optimized Python UDTF
  • SPARK-52946 Fix Arrow-optimized Python UDTF to support large var types
  • SPARK-52934 Allow yielding scalar values with Arrow-optimized Python UDTF
  • SPARK-52861 Skip Row object creation in Arrow-optimized UDTF execution
  • SPARK-51834 Sync OSS Support end-to-end create/replace table with constraint
  • SPARK-44856 Improve Python UDTF arrow serializer performance
  • SPARK-51834 Revert "[SQL] Fix TableSpec by adding constructor"
  • SPARK-51834 Fix TableSpec by adding constructor
  • SPARK-51834 Revert "[SQL] Sync OSS (partial): Support end-to-end table constraint in create/replace table"
  • SPARK-51834 Sync OSS (partial): Support end-to-end table constraint in create/replace table
  • SPARK-52885 Implement the hour, minute, and second functions in Scala for the TIME type
  • SPARK-51834 Sync OSS update ResolvedIdentifier unapply calls
  • SPARK-44856 Revert "[SC-195808][python] Improve Python UDTF arrow serializer performance"
  • SPARK-51834 Revert "[SQL] Sync OSS update ResolvedIdentifier unapply calls"
  • SPARK-44856 Improve Python UDTF arrow serializer performance
  • SPARK-51834 Sync OSS update ResolvedIdentifier unapply calls
  • SPARK-52787 Reorganize streaming execution dir around runtime and checkpoint areas
  • SPARK-51695 Sync OSS create/replace/alter table for unique constraint via DSv2
  • SPARK-51695 Sync OSS drop constraint via DSv2
  • SPARK-51695 Introduce Parser Changes for Table Constraints (CHECK, PK, FK)
  • SPARK-51695 Revert "[SQL] Introduce Parser Changes for Table Constraints (CHECK, PK, FK)"
  • SPARK-51695 Introduce Parser Changes for Table Constraints (CHECK, PK, FK)
  • SPARK-49428 Move Connect Scala Client from Connector to SQL
  • SPARK-52422 Update pandas to 2.3.0
  • SPARK-52228 Integrate state interaction microbenchmark into Quicksilver (TCP)
  • SPARK-52300 Make SQL UDTVF resolution use consistent configurations with view resolution
  • SPARK-52228 Construct the benchmark purposed TWS state server with in-memory state impls and the benchmark code in python
  • SPARK-52174 Enable spark.checkpoint.compress by default
  • SPARK-52224 Introduce pyyaml as a dependency for the Python client
  • SPARK-52122 Fix DefaultParamsReader RCE vulnerability
  • SPARK-51147 Refactor streaming related classes to a dedicated streaming directory
  • SPARK-51789 Respect spark.api.mode and spark.remote properly when parsing arguments in Spark Submission
  • SPARK-51212 Add a separated PySpark package for Spark Connect by default
  • SPARK-50762 Add Analyzer rule for resolving SQL scalar UDFs
  • SPARK-49748 Add getCondition and deprecate getErrorClass in SparkThrowable
  • SPARK-50605 Support SQL API mode for easier migration to Spark Connect
  • SPARK-50458 Proper error handling for unsupported file system when reading files
  • SPARK-49700 Unified Scala Interface for Connect and Classic
  • SPARK-51779 Use virtual column families for stream-stream joins
  • SPARK-51820 Move UnresolvedOrdinal construction before analysis to avoid issue with group by ordinal
  • SPARK-51814 Introduce a new API transformWithState in PySpark
  • SPARK-51635 Merge PushProjectionThroughLimit and PushProjectionThroughOffset
  • SPARK-51165 Enable spark.master.rest.enabled by default
  • SPARK-51688 Use Unix Domain Socket between Python and JVM communication
  • SPARK-51503 Support Variant type in XML scan
  • SPARK-50564 Upgrade protobuf Python package to 5.29.1
  • SPARK-50359 Upgrade PyArrow to 18.0
  • SPARK-51340 Model size estimation
  • SPARK-50657 Upgrade the minimum version of pyarrow to 11.0.0
  • SPARK-49282 Create a shared SparkSessionBuilder interface.
  • SPARK-51371 Change toString to toPrettySQL when building Aliases in ResolveAggregateFunctions
  • SPARK-50694 Support renames in subqueries
  • SPARK-50601 Revert "[SC-186342][sql] Support withColumns / withColumnsRenamed in subqueries"
  • SPARK-50601 Support withColumns / withColumnsRenamed in subqueries

Databricks ODBC/JDBC driver support​

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

Maintenance updates​

See Databricks Runtime maintenance updates.

System environment​

  • Operating System: Ubuntu 24.04.3 LTS
  • Java: Zulu21.42+19-CA
  • Scala: 2.13.16
  • Python: 3.12.3
  • R: 4.5.1
  • Delta Lake: 4.0.0

Installed Python libraries​

Library

Version

Library

Version

Library

Version

annotated-doc

0.0.3

annotated-types

0.7.0

anyio

4.7.0

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

arro3-core

0.6.5

arrow

1.3.0

asttokens

3.0.0

astunparse

1.6.3

async-lru

2.0.4

attrs

24.3.0

autocommand

2.2.2

azure-common

1.1.28

azure-core

1.36.0

azure-identity

1.20.0

azure-mgmt-core

1.6.0

azure-mgmt-web

8.0.0

azure-storage-blob

12.27.1

azure-storage-file-datalake

12.22.0

babel

2.16.0

backports.tarfile

1.2.0

beautifulsoup4

4.12.3

black

24.10.0

bleach

6.2.0

blinker

1.7.0

boto3

1.40.45

botocore

1.40.45

cachetools

5.5.1

certifi

2025.4.26

cffi

1.17.1

chardet

4.0.0

charset-normalizer

3.3.2

click

8.1.8

cloudpickle

3.0.0

comm

0.2.1

contourpy

1.3.1

cryptography

44.0.1

cycler

0.11.0

Cython

3.1.5

databricks-agents

1.4.0

databricks-connect

17.3.1

databricks-sdk

0.67.0

dataclasses-json

0.6.7

dbus-python

1.3.2

debugpy

1.8.11

decorator

5.1.1

defusedxml

0.7.1

deltalake

1.1.4

Deprecated

1.2.18

distlib

0.3.9

docstring-to-markdown

0.11

executing

1.2.0

facets-overview

1.1.1

fastapi

0.121.0

fastjsonschema

2.21.1

filelock

3.18.0

fonttools

4.55.3

fqdn

1.5.1

fsspec

2023.5.0

gitdb

4.0.11

GitPython

3.1.43

google-api-core

2.28.1

google-auth

2.43.0

google-cloud-core

2.5.0

google-cloud-storage

3.5.0

google-crc32c

1.7.1

google-resumable-media

2.7.2

googleapis-common-protos

1.65.0

grpcio

1.67.0

grpcio-status

1.67.0

h11

0.16.0

httpcore

1.0.9

httplib2

0.20.4

httpx

0.28.1

idna

3.7

importlib-metadata

6.6.0

inflect

7.3.1

iniconfig

1.1.1

ipyflow-core

0.0.209

ipykernel

6.29.5

ipython

8.30.0

ipython-genutils

0.2.0

ipywidgets

7.8.1

isodate

0.7.2

isoduration

20.11.0

jaraco.collections

5.1.0

jaraco.context

5.3.0

jaraco.functools

4.0.1

jaraco.text

3.12.1

jedi

0.19.2

Jinja2

3.1.6

jiter

0.11.1

jmespath

1.0.1

joblib

1.4.2

json5

0.9.25

jsonpatch

1.33

jsonpointer

3.0.0

jsonschema

4.23.0

jsonschema-specifications

2023.7.1

jupyter-events

0.12.0

jupyter-lsp

2.2.5

jupyter_client

8.6.3

jupyter_core

5.7.2

jupyter_server

2.15.0

jupyter_server_terminals

0.5.3

jupyterlab

4.3.4

jupyterlab_pygments

0.3.0

jupyterlab_server

2.27.3

jupyterlab_widgets

1.1.11

kiwisolver

1.4.8

langchain-core

1.0.3

langchain-openai

1.0.2

langsmith

0.4.41

launchpadlib

1.11.0

lazr.restfulclient

0.14.6

lazr.uri

1.0.6

markdown-it-py

2.2.0

MarkupSafe

3.0.2

marshmallow

3.26.1

matplotlib

3.10.0

matplotlib-inline

0.1.7

mccabe

0.7.0

mdurl

0.1.0

mistune

3.1.2

mlflow-skinny

3.5.1

mmh3

5.2.0

more-itertools

10.3.0

msal

1.34.0

msal-extensions

1.3.1

mypy-extensions

1.0.0

nbclient

0.10.2

nbconvert

7.16.6

nbformat

5.10.4

nest-asyncio

1.6.0

nodeenv

1.9.1

notebook

7.3.2

notebook_shim

0.2.4

numpy

2.1.3

oauthlib

3.2.2

openai

2.7.1

opentelemetry-api

1.38.0

opentelemetry-proto

1.38.0

opentelemetry-sdk

1.38.0

opentelemetry-semantic-conventions

0.59b0

orjson

3.11.4

overrides

7.4.0

packaging

24.2

pandas

2.2.3

pandocfilters

1.5.0

parso

0.8.4

pathspec

0.10.3

patsy

1.0.1

pexpect

4.8.0

pillow

11.1.0

pip

25.0.1

platformdirs

4.3.7

plotly

5.24.1

pluggy

1.5.0

prometheus_client

0.21.1

prompt-toolkit

3.0.43

proto-plus

1.26.1

protobuf

5.29.4

psutil

5.9.0

psycopg2

2.9.11

ptyprocess

0.7.0

pure-eval

0.2.2

py4j

0.10.9.9

pyarrow

21.0.0

pyasn1

0.4.8

pyasn1-modules

0.2.8

pyccolo

0.0.71

pycparser

2.21

pydantic

2.10.6

pydantic_core

2.27.2

pyflakes

3.2.0

Pygments

2.19.1

PyGObject

3.48.2

pyiceberg

0.10.0

PyJWT

2.10.1

pyodbc

5.2.0

pyparsing

3.2.0

pyright

1.1.394

pyroaring

1.0.3

pytest

8.3.5

python-dateutil

2.9.0.post0

python-dotenv

1.2.1

python-json-logger

3.2.1

python-lsp-jsonrpc

1.1.2

python-lsp-server

1.12.2

pytoolconfig

1.2.6

pytz

2024.1

PyYAML

6.0.2

pyzmq

26.2.0

referencing

0.30.2

regex

2024.11.6

requests

2.32.3

requests-toolbelt

1.0.0

rfc3339-validator

0.1.4

rfc3986-validator

0.1.1

rich

13.9.4

rope

1.13.0

rpds-py

0.22.3

rsa

4.9.1

s3transfer

0.14.0

scikit-learn

1.6.1

scipy

1.15.3

seaborn

0.13.2

Send2Trash

1.8.2

setuptools

78.1.1

six

1.17.0

smmap

5.0.0

sniffio

1.3.0

sortedcontainers

2.4.0

soupsieve

2.5

sqlparse

0.5.3

ssh-import-id

5.11

stack-data

0.6.3

starlette

0.49.3

strictyaml

1.7.3

tenacity

9.0.0

terminado

0.17.1

threadpoolctl

3.5.0

tiktoken

0.12.0

tinycss2

1.4.0

tokenize_rt

6.1.0

tomli

2.0.1

tornado

6.5.1

tqdm

4.67.1

traitlets

5.14.3

typeguard

4.3.0

types-python-dateutil

2.9.0.20251008

typing-inspect

0.9.0

typing_extensions

4.12.2

tzdata

2024.1

ujson

5.10.0

unattended-upgrades

0.1

uri-template

1.3.0

urllib3

2.3.0

uvicorn

0.38.0

virtualenv

20.29.3

wadllib

1.3.6

wcwidth

0.2.5

webcolors

25.10.0

webencodings

0.5.1

websocket-client

1.8.0

whatthepatch

1.0.2

wheel

0.45.1

whenever

0.7.3

widgetsnbextension

3.6.6

wrapt

1.17.0

yapf

0.40.2

zipp

3.21.0

zstandard

0.23.0

Installed R libraries​

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20.

Library

Version

Library

Version

Library

Version

arrow

22.0.0

askpass

1.2.1

assertthat

0.2.1

backports

1.5.0

base

4.5.1

base64enc

0.1-3

bigD

0.3.1

bit

4.6.0

bit64

4.6.0-1

bitops

1.0-9

blob

1.2.4

boot

1.3-30

brew

1.0-10

brio

1.1.5

broom

1.0.10

bslib

0.9.0

cachem

1.1.0

callr

3.7.6

caret

7.0-1

cellranger

1.1.0

chron

2.3-62

class

7.3-22

cli

3.6.5

clipr

0.8.0

clock

0.7.3

cluster

2.1.6

codetools

0.2-20

commonmark

2.0.0

compiler

4.5.1

config

0.3.2

conflicted

1.2.0

cpp11

0.5.2

crayon

1.5.3

credentials

2.0.3

curl

7.0.0

data.table

1.17.8

datasets

4.5.1

DBI

1.2.3

dbplyr

2.5.1

desc

1.4.3

devtools

2.4.6

diagram

1.6.5

diffobj

0.3.6

digest

0.6.39

downlit

0.4.5

dplyr

1.1.4

dtplyr

1.3.2

e1071

1.7-16

ellipsis

0.3.2

evaluate

1.0.5

fansi

1.0.7

farver

2.1.2

fastmap

1.2.0

fontawesome

0.5.3

forcats

1.0.1

foreach

1.5.2

foreign

0.8-86

fs

1.6.6

future

1.68.0

future.apply

1.20.0

gargle

1.6.0

generics

0.1.4

gert

2.2.0

ggplot2

4.0.1

gh

1.5.0

git2r

0.36.2

gitcreds

0.1.2

glmnet

4.1-10

globals

0.18.0

glue

1.8.0

googledrive

2.1.2

googlesheets4

1.1.2

gower

1.0.2

graphics

4.5.1

grDevices

4.5.1

grid

4.5.1

gridExtra

2.3

gsubfn

0.7

gt

1.1.0

gtable

0.3.6

hardhat

1.4.2

haven

2.5.5

highr

0.11

hms

1.1.4

htmltools

0.5.8.1

htmlwidgets

1.6.4

httpuv

1.6.16

httr

1.4.7

httr2

1.2.1

ids

1.0.1

ini

0.3.1

ipred

0.9-15

isoband

0.2.7

iterators

1.0.14

jquerylib

0.1.4

jsonlite

2.0.0

juicyjuice

0.1.0

KernSmooth

2.23-22

knitr

1.50

labeling

0.4.3

later

1.4.4

lattice

0.22-5

lava

1.8.2

lifecycle

1.0.4

listenv

0.10.0

litedown

0.8

lubridate

1.9.4

magrittr

2.0.4

markdown

2.0

MASS

7.3-60.0.1

Matrix

1.6-5

memoise

2.0.1

methods

4.5.1

mgcv

1.9-1

mime

0.13

miniUI

0.1.2

mlflow

3.6.0

ModelMetrics

1.2.2.2

modelr

0.1.11

nlme

3.1-164

nnet

7.3-19

numDeriv

2016.8-1.1

openssl

2.3.4

otel

0.2.0

parallel

4.5.1

parallelly

1.45.1

pillar

1.11.1

pkgbuild

1.4.8

pkgconfig

2.0.3

pkgdown

2.2.0

pkgload

1.4.1

plogr

0.2.0

plyr

1.8.9

praise

1.0.0

prettyunits

1.2.0

pROC

1.19.0.1

processx

3.8.6

prodlim

2025.04.28

profvis

0.4.0

progress

1.2.3

progressr

0.18.0

promises

1.5.0

proto

1.0.0

proxy

0.4-27

ps

1.9.1

purrr

1.2.0

R6

2.6.1

ragg

1.5.0

randomForest

4.7-1.2

rappdirs

0.3.3

rcmdcheck

1.4.0

RColorBrewer

1.1-3

Rcpp

1.1.0

RcppEigen

0.3.4.0.2

reactable

0.4.4

reactR

0.6.1

readr

2.1.6

readxl

1.4.5

recipes

1.3.1

rematch

2.0.0

rematch2

2.1.2

remotes

2.5.0

reprex

2.1.1

reshape2

1.4.5

rlang

1.1.6

rmarkdown

2.30

RODBC

1.3-26

roxygen2

7.3.3

rpart

4.1.23

rprojroot

2.1.1

Rserve

1.8-15

RSQLite

2.4.4

rstudioapi

0.17.1

rversions

3.0.0

rvest

1.0.5

S7

0.2.1

sass

0.4.10

scales

1.4.0

selectr

0.4-2

sessioninfo

1.2.3

shape

1.4.6.1

shiny

1.11.1

sourcetools

0.1.7-1

sparklyr

1.9.3

SparkR

4.1.0

sparsevctrs

0.3.4

spatial

7.3-17

splines

4.5.1

sqldf

0.4-11

SQUAREM

2021.1

stats

4.5.1

stats4

4.5.1

stringi

1.8.7

stringr

1.6.0

survival

3.5-8

swagger

5.17.14.1

sys

3.4.3

systemfonts

1.3.1

tcltk

4.5.1

testthat

3.3.0

textshaping

1.0.4

tibble

3.3.0

tidyr

1.3.1

tidyselect

1.2.1

tidyverse

2.0.0

timechange

0.3.0

timeDate

4051.111

tinytex

0.58

tools

4.5.1

tzdb

0.5.0

urlchecker

1.0.1

usethis

3.2.1

utf8

1.2.6

utils

4.5.1

uuid

1.2-1

V8

8.0.1

vctrs

0.6.5

viridisLite

0.4.2

vroom

1.6.6

waldo

0.6.2

whisker

0.4.1

withr

3.0.2

xfun

0.54

xml2

1.5.0

xopen

1.0.1

xtable

1.8-4

yaml

2.3.10

zeallot

0.2.0

zip

2.3.3

Installed Java and Scala libraries (Scala 2.13 cluster version)​

Group ID

Artifact ID

Version

antlr

antlr

2.7.7

com.amazonaws

amazon-kinesis-client

1.15.3

com.amazonaws

aws-java-sdk-autoscaling

1.12.681

com.amazonaws

aws-java-sdk-cloudformation

1.12.681

com.amazonaws

aws-java-sdk-cloudfront

1.12.681

com.amazonaws

aws-java-sdk-cloudhsm

1.12.681

com.amazonaws

aws-java-sdk-cloudsearch

1.12.681

com.amazonaws

aws-java-sdk-cloudtrail

1.12.681

com.amazonaws

aws-java-sdk-cloudwatch

1.12.681

com.amazonaws

aws-java-sdk-cloudwatchmetrics

1.12.681

com.amazonaws

aws-java-sdk-codedeploy

1.12.681

com.amazonaws

aws-java-sdk-cognitoidentity

1.12.681

com.amazonaws

aws-java-sdk-cognitosync

1.12.681

com.amazonaws

aws-java-sdk-config

1.12.681

com.amazonaws

aws-java-sdk-core

1.12.681

com.amazonaws

aws-java-sdk-datapipeline

1.12.681

com.amazonaws

aws-java-sdk-directconnect

1.12.681

com.amazonaws

aws-java-sdk-directory

1.12.681

com.amazonaws

aws-java-sdk-dynamodb

1.12.681

com.amazonaws

aws-java-sdk-ec2

1.12.681

com.amazonaws

aws-java-sdk-ecs

1.12.681

com.amazonaws

aws-java-sdk-efs

1.12.681

com.amazonaws

aws-java-sdk-elasticache

1.12.681

com.amazonaws

aws-java-sdk-elasticbeanstalk

1.12.681

com.amazonaws

aws-java-sdk-elasticloadbalancing

1.12.681

com.amazonaws

aws-java-sdk-elastictranscoder

1.12.681

com.amazonaws

aws-java-sdk-emr

1.12.681

com.amazonaws

aws-java-sdk-glacier

1.12.681

com.amazonaws

aws-java-sdk-glue

1.12.681

com.amazonaws

aws-java-sdk-iam

1.12.681

com.amazonaws

aws-java-sdk-importexport

1.12.681

com.amazonaws

aws-java-sdk-kinesis

1.12.681

com.amazonaws

aws-java-sdk-kms

1.12.681

com.amazonaws

aws-java-sdk-lambda

1.12.681

com.amazonaws

aws-java-sdk-logs

1.12.681

com.amazonaws

aws-java-sdk-machinelearning

1.12.681

com.amazonaws

aws-java-sdk-opsworks

1.12.681

com.amazonaws

aws-java-sdk-rds

1.12.681

com.amazonaws

aws-java-sdk-redshift

1.12.681

com.amazonaws

aws-java-sdk-route53

1.12.681

com.amazonaws

aws-java-sdk-s3

1.12.681

com.amazonaws

aws-java-sdk-ses

1.12.681

com.amazonaws

aws-java-sdk-simpledb

1.12.681

com.amazonaws

aws-java-sdk-simpleworkflow

1.12.681

com.amazonaws

aws-java-sdk-sns

1.12.681

com.amazonaws

aws-java-sdk-sqs

1.12.681

com.amazonaws

aws-java-sdk-ssm

1.12.681

com.amazonaws

aws-java-sdk-storagegateway

1.12.681

com.amazonaws

aws-java-sdk-sts

1.12.681

com.amazonaws

aws-java-sdk-support

1.12.681

com.amazonaws

aws-java-sdk-workspaces

1.12.681

com.amazonaws

jmespath-java

1.12.681

com.clearspring.analytics

stream

2.9.8

com.databricks

Rserve

1.8-3

com.databricks

databricks-sdk-java

0.53.0

com.databricks

jets3t

0.7.1-0

com.databricks.scalapb

scalapb-runtime_2.13

0.4.15-11

com.esotericsoftware

kryo-shaded

4.0.3

com.esotericsoftware

minlog

1.3.0

com.fasterxml

classmate

1.5.1

com.fasterxml.jackson.core

jackson-annotations

2.18.2

com.fasterxml.jackson.core

jackson-core

2.18.2

com.fasterxml.jackson.core

jackson-databind

2.18.2

com.fasterxml.jackson.dataformat

jackson-dataformat-cbor

2.18.2

com.fasterxml.jackson.dataformat

jackson-dataformat-yaml

2.15.2

com.fasterxml.jackson.datatype

jackson-datatype-joda

2.18.2

com.fasterxml.jackson.datatype

jackson-datatype-jsr310

2.18.3

com.fasterxml.jackson.module

jackson-module-paranamer

2.18.2

com.fasterxml.jackson.module

jackson-module-scala_2.13

2.18.2

com.github.ben-manes.caffeine

caffeine

2.9.3

com.github.blemale

scaffeine_2.13

4.1.0

com.github.fommil

jniloader

1.1

com.github.fommil.netlib

native_ref-java

1.1

com.github.fommil.netlib

native_ref-java

1.1-natives

com.github.fommil.netlib

native_system-java

1.1

com.github.fommil.netlib

native_system-java

1.1-natives

com.github.fommil.netlib

netlib-native_ref-linux-x86_64

1.1-natives

com.github.fommil.netlib

netlib-native_system-linux-x86_64

1.1-natives

com.github.luben

zstd-jni

1.5.7-6

com.github.virtuald

curvesapi

1.08

com.github.wendykierp

JTransforms

3.1

com.google.api.grpc

proto-google-common-protos

2.5.1

com.google.auth

google-auth-library-credentials

1.20.0

com.google.auth

google-auth-library-oauth2-http

1.20.0

com.google.auto.value

auto-value-annotations

1.10.4

com.google.code.findbugs

jsr305

3.0.0

com.google.code.gson

gson

2.11.0

com.google.crypto.tink

tink

1.16.0

com.google.errorprone

error_prone_annotations

2.36.0

com.google.flatbuffers

flatbuffers-java

25.2.10

com.google.guava

failureaccess

1.0.3

com.google.guava

guava

33.4.8-jre

com.google.http-client

google-http-client

1.43.3

com.google.http-client

google-http-client-gson

1.43.3

com.google.j2objc

j2objc-annotations

3.0.0

com.google.protobuf

protobuf-java

3.25.5

com.google.protobuf

protobuf-java-util

3.25.5

com.helger

profiler

1.1.1

com.ibm.icu

icu4j

75.1

com.jcraft

jsch

0.1.55

com.lihaoyi

sourcecode_2.13

0.1.9

com.microsoft.azure

azure-data-lake-store-sdk

2.3.10

com.microsoft.sqlserver

mssql-jdbc

12.8.0.jre11

com.microsoft.sqlserver

mssql-jdbc

12.8.0.jre8

com.ning

compress-lzf

1.1.2

com.sun.mail

javax.mail

1.5.2

com.sun.xml.bind

jaxb-core

2.2.11

com.sun.xml.bind

jaxb-impl

2.2.11

com.tdunning

json

1.8

com.thoughtworks.paranamer

paranamer

2.8

com.trueaccord.lenses

lenses_2.13

0.4.13

com.twitter

chill-java

0.10.0

com.twitter

chill_2.13

0.10.0

com.twitter

util-app_2.13

19.8.1

com.twitter

util-core_2.13

19.8.1

com.twitter

util-function_2.13

19.8.1

com.twitter

util-jvm_2.13

19.8.1

com.twitter

util-lint_2.13

19.8.1

com.twitter

util-registry_2.13

19.8.1

com.twitter

util-stats_2.13

19.8.1

com.typesafe

config

1.4.3

com.typesafe.scala-logging

scala-logging_2.13

3.9.2

com.uber

h3

3.7.3

com.univocity

univocity-parsers

2.9.1

com.zaxxer

HikariCP

4.0.3

com.zaxxer

SparseBitSet

1.3

commons-cli

commons-cli

1.10.0

commons-codec

commons-codec

1.19.0

commons-collections

commons-collections

3.2.2

commons-dbcp

commons-dbcp

1.4

commons-fileupload

commons-fileupload

1.6.0

commons-httpclient

commons-httpclient

3.1

commons-io

commons-io

2.21.0

commons-lang

commons-lang

2.6

commons-logging

commons-logging

1.1.3

commons-pool

commons-pool

1.5.4

dev.ludovic.netlib

arpack

3.0.4

dev.ludovic.netlib

blas

3.0.4

dev.ludovic.netlib

lapack

3.0.4

info.ganglia.gmetric4j

gmetric4j

1.0.10

io.airlift

aircompressor

2.0.2

io.delta

delta-sharing-client_2.13

1.3.6

io.dropwizard.metrics

metrics-annotation

4.2.37

io.dropwizard.metrics

metrics-core

4.2.37

io.dropwizard.metrics

metrics-graphite

4.2.37

io.dropwizard.metrics

metrics-healthchecks

4.2.37

io.dropwizard.metrics

metrics-jetty10

4.2.37

io.dropwizard.metrics

metrics-jmx

4.2.37

io.dropwizard.metrics

metrics-json

4.2.37

io.dropwizard.metrics

metrics-jvm

4.2.37

io.dropwizard.metrics

metrics-servlets

4.2.37

io.github.java-diff-utils

java-diff-utils

4.15

io.netty

netty-all

4.2.7.Final

io.netty

netty-buffer

4.2.7.Final

io.netty

netty-codec

4.2.7.Final

io.netty

netty-codec-base

4.2.7.Final

io.netty

netty-codec-classes-quic

4.2.7.Final

io.netty

netty-codec-compression

4.2.7.Final

io.netty

netty-codec-http

4.2.7.Final

io.netty

netty-codec-http2

4.2.7.Final

io.netty

netty-codec-http3

4.2.7.Final

io.netty

netty-codec-marshalling

4.2.7.Final

io.netty

netty-codec-native-quic

4.2.7.Final-linux-aarch_64

io.netty

netty-codec-native-quic

4.2.7.Final-linux-x86_64

io.netty

netty-codec-native-quic

4.2.7.Final-osx-aarch_64

io.netty

netty-codec-native-quic

4.2.7.Final-osx-x86_64

io.netty

netty-codec-native-quic

4.2.7.Final-windows-x86_64

io.netty

netty-codec-protobuf

4.2.7.Final

io.netty

netty-codec-socks

4.2.7.Final

io.netty

netty-common

4.2.7.Final

io.netty

netty-handler

4.2.7.Final

io.netty

netty-handler-proxy

4.2.7.Final

io.netty

netty-resolver

4.2.7.Final

io.netty

netty-tcnative-boringssl-static

2.0.74.Final-db-r0-linux-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.74.Final-db-r0-linux-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.74.Final-db-r0-osx-aarch_64

io.netty

netty-tcnative-boringssl-static

2.0.74.Final-db-r0-osx-x86_64

io.netty

netty-tcnative-boringssl-static

2.0.74.Final-db-r0-windows-x86_64

io.netty

netty-tcnative-classes

2.0.74.Final

io.netty

netty-transport

4.2.7.Final

io.netty

netty-transport-classes-epoll

4.2.7.Final

io.netty

netty-transport-classes-io_uring

4.2.7.Final

io.netty

netty-transport-classes-kqueue

4.2.7.Final

io.netty

netty-transport-native-epoll

4.2.7.Final

io.netty

netty-transport-native-epoll

4.2.7.Final-linux-aarch_64

io.netty

netty-transport-native-epoll

4.2.7.Final-linux-riscv64

io.netty

netty-transport-native-epoll

4.2.7.Final-linux-x86_64

io.netty

netty-transport-native-io_uring

4.2.7.Final-linux-aarch_64

io.netty

netty-transport-native-io_uring

4.2.7.Final-linux-riscv64

io.netty

netty-transport-native-io_uring

4.2.7.Final-linux-x86_64

io.netty

netty-transport-native-kqueue

4.2.7.Final-osx-aarch_64

io.netty

netty-transport-native-kqueue

4.2.7.Final-osx-x86_64

io.netty

netty-transport-native-unix-common

4.2.7.Final

io.opencensus

opencensus-api

0.31.1

io.opencensus

opencensus-contrib-http-util

0.31.1

io.prometheus

simpleclient

0.16.1-databricks

io.prometheus

simpleclient_common

0.16.1-databricks

io.prometheus

simpleclient_dropwizard

0.16.1-databricks

io.prometheus

simpleclient_pushgateway

0.16.1-databricks

io.prometheus

simpleclient_servlet

0.16.1-databricks

io.prometheus

simpleclient_servlet_common

0.16.1-databricks

io.prometheus

simpleclient_tracer_common

0.16.1-databricks

io.prometheus

simpleclient_tracer_otel

0.16.1-databricks

io.prometheus

simpleclient_tracer_otel_agent

0.16.1-databricks

io.prometheus.jmx

collector

0.18.0

jakarta.annotation

jakarta.annotation-api

1.3.5

jakarta.servlet

jakarta.servlet-api

4.0.3

jakarta.validation

jakarta.validation-api

2.0.2

jakarta.ws.rs

jakarta.ws.rs-api

2.1.6

javax.activation

activation

1.1.1

javax.annotation

javax.annotation-api

1.3.2

javax.el

javax.el-api

2.2.4

javax.jdo

jdo-api

3.0.1

javax.media

jai_core

jai_core_dummy

javax.transaction

jta

1.1

javax.transaction

transaction-api

1.1

javax.xml.bind

jaxb-api

2.2.11

javolution

javolution

5.5.1

jline

jline

2.14.6

joda-time

joda-time

2.14.0

net.java.dev.jna

jna

5.8.0

net.razorvine

pickle

1.5

net.sf.jpam

jpam

1.1

net.sf.opencsv

opencsv

2.3

net.sf.supercsv

super-csv

2.2.0

net.snowflake

snowflake-ingest-sdk

0.9.6

net.sourceforge.f2j

arpack_combined_all

0.1

org.acplt.remotetea

remotetea-oncrpc

1.1.2

org.antlr

ST4

4.0.4

org.antlr

antlr-runtime

3.5.2

org.antlr

antlr4-runtime

4.13.1

org.antlr

stringtemplate

3.2.1

org.apache.ant

ant

1.10.11

org.apache.ant

ant-jsch

1.10.11

org.apache.ant

ant-launcher

1.10.11

org.apache.arrow

arrow-format

18.3.0

org.apache.arrow

arrow-memory-core

18.3.0

org.apache.arrow

arrow-memory-netty

18.3.0

org.apache.arrow

arrow-memory-netty-buffer-patch

18.3.0

org.apache.arrow

arrow-vector

18.3.0

org.apache.avro

avro

1.12.1

org.apache.avro

avro-ipc

1.12.1

org.apache.avro

avro-mapred

1.12.1

org.apache.commons

commons-collections4

4.5.0

org.apache.commons

commons-compress

1.28.0

org.apache.commons

commons-configuration2

2.11.0

org.apache.commons

commons-crypto

1.1.0

org.apache.commons

commons-lang3

3.19.0

org.apache.commons

commons-math3

3.6.1

org.apache.commons

commons-text

1.14.0

org.apache.curator

curator-client

5.9.0

org.apache.curator

curator-framework

5.9.0

org.apache.curator

curator-recipes

5.9.0

org.apache.datasketches

datasketches-java

6.2.0

org.apache.datasketches

datasketches-memory

3.0.2

org.apache.derby

derby

10.14.2.0

org.apache.hadoop

hadoop-client-runtime

3.4.2

org.apache.hive

hive-beeline

2.3.10

org.apache.hive

hive-cli

2.3.10

org.apache.hive

hive-jdbc

2.3.10

org.apache.hive

hive-llap-client

2.3.10

org.apache.hive

hive-llap-common

2.3.10

org.apache.hive

hive-serde

2.3.10

org.apache.hive

hive-shims

2.3.10

org.apache.hive

hive-storage-api

2.8.1

org.apache.hive.shims

hive-shims-0.23

2.3.10

org.apache.hive.shims

hive-shims-common

2.3.10

org.apache.hive.shims

hive-shims-scheduler

2.3.10

org.apache.httpcomponents

httpclient

4.5.14

org.apache.httpcomponents

httpcore

4.4.16

org.apache.ivy

ivy

2.5.3

org.apache.logging.log4j

log4j-1.2-api

2.24.3

org.apache.logging.log4j

log4j-api

2.24.3

org.apache.logging.log4j

log4j-core

2.24.3

org.apache.logging.log4j

log4j-layout-template-json

2.24.3

org.apache.logging.log4j

log4j-slf4j2-impl

2.24.3

org.apache.orc

orc-core

2.2.0-shaded-protobuf

org.apache.orc

orc-format

1.1.1-shaded-protobuf

org.apache.orc

orc-mapreduce

2.2.0-shaded-protobuf

org.apache.orc

orc-shims

2.2.0

org.apache.poi

poi

5.4.1

org.apache.poi

poi-ooxml

5.4.1

org.apache.poi

poi-ooxml-full

5.4.1

org.apache.poi

poi-ooxml-lite

5.4.1

org.apache.thrift

libfb303

0.9.3

org.apache.thrift

libthrift

0.16.0

org.apache.ws.xmlschema

xmlschema-core

2.3.1

org.apache.xbean

xbean-asm9-shaded

4.28

org.apache.xmlbeans

xmlbeans

5.3.0

org.apache.yetus

audience-annotations

0.13.0

org.apache.zookeeper

zookeeper

3.9.4

org.apache.zookeeper

zookeeper-jute

3.9.4

org.checkerframework

checker-qual

3.43.0

org.codehaus.janino

commons-compiler

3.0.16

org.codehaus.janino

janino

3.0.16

org.datanucleus

datanucleus-api-jdo

4.2.4

org.datanucleus

datanucleus-core

4.1.17

org.datanucleus

datanucleus-rdbms

4.1.19

org.datanucleus

javax.jdo

3.2.0-m3

org.eclipse.jetty

jetty-alpn-client

10.0.26

org.eclipse.jetty

jetty-client

10.0.26

org.eclipse.jetty

jetty-http

10.0.26

org.eclipse.jetty

jetty-io

10.0.26

org.eclipse.jetty

jetty-jndi

10.0.26

org.eclipse.jetty

jetty-plus

10.0.26

org.eclipse.jetty

jetty-proxy

10.0.26

org.eclipse.jetty

jetty-security

10.0.26

org.eclipse.jetty

jetty-server

10.0.26

org.eclipse.jetty

jetty-servlet

10.0.26

org.eclipse.jetty

jetty-servlets

10.0.26

org.eclipse.jetty

jetty-util

10.0.26

org.eclipse.jetty

jetty-webapp

10.0.26

org.eclipse.jetty

jetty-xml

10.0.26

org.fusesource.leveldbjni

leveldbjni-all

1.8

org.glassfish.hk2

hk2-api

2.6.1

org.glassfish.hk2

hk2-locator

2.6.1

org.glassfish.hk2

hk2-utils

2.6.1

org.glassfish.hk2

osgi-resource-locator

1.0.3

org.glassfish.hk2.external

aopalliance-repackaged

2.6.1

org.glassfish.hk2.external

jakarta.inject

2.6.1

org.glassfish.jersey.containers

jersey-container-servlet

2.41

org.glassfish.jersey.containers

jersey-container-servlet-core

2.41

org.glassfish.jersey.core

jersey-client

2.41

org.glassfish.jersey.core

jersey-common

2.41

org.glassfish.jersey.core

jersey-server

2.41

org.glassfish.jersey.inject

jersey-hk2

2.41

org.hibernate.validator

hibernate-validator

6.2.5.Final

org.ini4j

ini4j

0.5.4

org.javassist

javassist

3.29.2-GA

org.jboss.logging

jboss-logging

3.4.1.Final

org.jdbi

jdbi

2.63.1

org.jline

jline

3.27.1-jdk8

org.joda

joda-convert

1.7

org.json4s

json4s-ast_2.13

4.0.7

org.json4s

json4s-core_2.13

4.0.7

org.json4s

json4s-jackson-core_2.13

4.0.7

org.json4s

json4s-jackson_2.13

4.0.7

org.json4s

json4s-scalap_2.13

4.0.7

org.locationtech.jts

jts-core

1.20.0

org.lz4

lz4-java

1.8.0-databricks-1

org.mlflow

mlflow-spark_2.13

2.22.1

org.objenesis

objenesis

3.4

org.postgresql

postgresql

42.6.1

org.roaringbitmap

RoaringBitmap

1.2.1

org.rocksdb

rocksdbjni

9.8.4

org.rosuda.REngine

REngine

2.1.0

org.scala-lang

scala-compiler_2.13

2.13.16

org.scala-lang

scala-library_2.13

2.13.16

org.scala-lang

scala-reflect_2.13

2.13.16

org.scala-lang.modules

scala-collection-compat_2.13

2.11.0

org.scala-lang.modules

scala-java8-compat_2.13

0.9.1

org.scala-lang.modules

scala-parallel-collections_2.13

1.2.0

org.scala-lang.modules

scala-parser-combinators_2.13

2.4.0

org.scala-lang.modules

scala-xml_2.13

2.4.0

org.scala-sbt

test-interface

1.0

org.scalacheck

scalacheck_2.13

1.18.0

org.scalactic

scalactic_2.13

3.2.19

org.scalanlp

breeze-macros_2.13

2.1.0

org.scalanlp

breeze_2.13

2.1.0

org.scalatest

scalatest-compatible

3.2.19

org.scalatest

scalatest-core_2.13

3.2.19

org.scalatest

scalatest-diagrams_2.13

3.2.19

org.scalatest

scalatest-featurespec_2.13

3.2.19

org.scalatest

scalatest-flatspec_2.13

3.2.19

org.scalatest

scalatest-freespec_2.13

3.2.19

org.scalatest

scalatest-funspec_2.13

3.2.19

org.scalatest

scalatest-funsuite_2.13

3.2.19

org.scalatest

scalatest-matchers-core_2.13

3.2.19

org.scalatest

scalatest-mustmatchers_2.13

3.2.19

org.scalatest

scalatest-propspec_2.13

3.2.19

org.scalatest

scalatest-refspec_2.13

3.2.19

org.scalatest

scalatest-shouldmatchers_2.13

3.2.19

org.scalatest

scalatest-wordspec_2.13

3.2.19

org.scalatest

scalatest_2.13

3.2.19

org.slf4j

jcl-over-slf4j

2.0.16

org.slf4j

jul-to-slf4j

2.0.16

org.slf4j

slf4j-api

2.0.16

org.slf4j

slf4j-simple

1.7.25

org.threeten

threeten-extra

1.8.0

org.tukaani

xz

1.10

org.typelevel

algebra_2.13

2.8.0

org.typelevel

cats-kernel_2.13

2.8.0

org.typelevel

spire-macros_2.13

0.18.0

org.typelevel

spire-platform_2.13

0.18.0

org.typelevel

spire-util_2.13

0.18.0

org.typelevel

spire_2.13

0.18.0

org.wildfly.openssl

wildfly-openssl

1.1.3.Final

org.xerial

sqlite-jdbc

3.42.0.0

org.xerial.snappy

snappy-java

1.1.10.3

org.yaml

snakeyaml

2.0

oro

oro

2.0.8

pl.edu.icm

JLargeArrays

1.5

software.amazon.cryptools

AmazonCorrettoCryptoProvider

2.5.0-linux-x86_64

stax

stax-api

1.0.1

ヒント

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.