Skip to main content
RSS Feed
Last updated on

Serverless compute release notes

Applies to: check marked yes AWS check marked yes GCP check marked yes Azure

This section includes release notes for serverless compute. Release notes are organized by year and week of year. Serverless compute always runs using the most recently released version listed here.

July 6, 2026​

This serverless compute release includes updates from SAP Databricks Runtime 18.

New features​

  • IP address functions (Preview): New SQL functions are available for working with IPv4 and IPv6 addresses and CIDR blocks, including ip_host, ip_cidr, ip_version, ip_prefix_length, ip_network, ip_network_last, ip_cidr_contains, ip_as_binary, ip_as_string, and try_* variants for null-safe behavior.

  • On-demand state repartitioning (Preview): Structured Streaming now supports changing the number of shuffle partitions for stateful queries without losing checkpoint state.

Behavior changes​

  • CREATE OR REPLACE TABLE preserves comments: CREATE OR REPLACE TABLE now preserves existing column and table comments by default. Previously, comments were dropped when recreating a table.

  • DataFrame by-name writes cast compatible columns: writeTo().append(), writeTo().overwrite(), writeTo().overwritePartitions(), and write.mode("append").saveAsTable() now automatically cast type-compatible columns (for example, int to long) to match the target Delta table schema. Previously, these operations failed with a DELTA_FAILED_TO_MERGE_FIELDS error when column types were compatible but not identical. Behavior now matches SQL INSERT INTO ... BY NAME.

  • ALTER TABLE SET TBLPROPERTIES for pipelines.pipelineId: ALTER TABLE <table> SET TBLPROPERTIES('pipelines.pipelineId' = '<pipeline-id>') now attempts to make the specified table eligible for writes by the pipeline. Previously, setting this property on a regular table had no effect. If the table isn't eligible for pipeline writes, the command throws SETTING_PIPELINES_PIPELINE_ID_NOT_SUPPORTED.

  • DESCRIBE EXTENDED AS JSON includes predictive optimization results: DESCRIBE EXTENDED ... AS JSON now includes predictive optimization evaluation results in its output. Previously, this information wasn't returned in the JSON output.

  • Metric view window measures return correct results: Metric view window measures now return correct results when queries use GROUP BY, IN/BETWEEN filters, or mixed predicates on the window's order column. Previously, these filter patterns could produce incorrect results.

  • Structured Streaming deduplication with NaN keys: Structured Streaming deduplication now treats NaN (Not-a-Number) values that have different bit patterns as duplicates when a double or float column is used as a deduplication key. Previously, NaN values with different internal representations were treated as distinct and were not deduplicated.

  • NATURAL JOIN case-insensitive column matching: NATURAL JOIN now matches common columns case-insensitively, consistent with the equivalent USING join. Previously, column matching was case-sensitive, causing columns that differ only in case (for example, ID vs id) to not be recognized as common columns, resulting in a silent cross join instead of the expected equi-join.

Version 18.2​

May 13, 2026

This serverless compute release roughly corresponds to SAP Databricks Runtime 18.2.

New features​

  • CREATE OR REPLACE support for temporary tables: CREATE OR REPLACE TEMP TABLE syntax is now supported, allowing you to create or replace temporary tables in a single statement. This eliminates the need to explicitly drop and recreate temporary tables.

  • agg() alias for measure() function: agg() is now available as an alias for the measure() function. This change is fully backward compatible. Existing queries that use measure() continue to work without modification, and agg() produces identical results when used with the same arguments.

  • Delta table history includes write option flags: Delta table history (DESCRIBE HISTORY) now includes write option flags in the operationParameters column for WRITE and REPLACE TABLE operations. When the following options are explicitly enabled, they appear as boolean flags in the history (only included when true):

    For WRITE and REPLACE TABLE operations:

    • isDynamicPartitionOverwrite: present when dynamic partition overwrite mode was used
    • canOverwriteSchema: present when schema overwrite (overwriteSchema) was enabled
    • canMergeSchema: present when schema merge (mergeSchema) was enabled

    For REPLACE TABLE operations:

    • predicate: present when replaceWhere was used
    • isV1WriterSaveAsTableOverwrite: present when the replace was triggered by a .saveAsTable overwrite

  • Selectively replace data with replaceOn and replaceUsing DataFrame APIs: The replaceOn and replaceUsing options in the Scala and Python DataFrame APIs are now generally available. Use these options to replace part of the table with the result of a DataFrame. replaceOn replaces rows that match a user-defined condition. replaceUsing replaces rows where specified columns are equal. These APIs complement the INSERT REPLACE ON and INSERT REPLACE USING SQL statements.

Behavior changes​

  • NULL struct preservation in INSERT, MERGE, and streaming writes with schema evolution: For INSERT, MERGE, and streaming writes that use schema evolution, a NULL struct in the source is now stored as NULL in the target. Previously, that value was incorrectly materialized as a non-null struct with every field set to NULL, while the same operations without schema evolution preserved NULL structs correctly. If your code relied on receiving a non-null struct whose fields were all NULL, update your code to handle a NULL struct instead.

  • Fix for LEFT OUTER JOIN LATERAL dropping rows: A bug that incorrectly dropped rows from LEFT OUTER JOIN LATERAL queries is now fixed. Queries using this construct now return the correct results. To temporarily revert to the previous behavior, set spark.databricks.sql.optimizer.lateralJoinPreserveOuterSemantic to true.

  • NATURAL JOIN respects case-insensitive column matching: NATURAL JOIN now correctly uses case-insensitive column matching when spark.sql.caseSensitive is set to false (the default). Previously, NATURAL JOIN used case-sensitive comparison to identify common columns, causing columns that differed only in case (for example, ID versus id) to not be recognized as matching. This caused NATURAL JOIN to silently produce cross-join results. This fix aligns NATURAL JOIN behavior with USING joins, which already handled case-insensitivity correctly. Queries affected by this bug now return correct results with properly joined columns.

  • SQL UDF dependency validation in Unity Catalog: Unity Catalog now enforces dependency validation for SQL user-defined functions (UDFs) to prevent access control bypass. Previously, SQL functions created through the REST API could reference dependencies the user did not have access to. SQL UDFs with invalid dependency configurations are now blocked from execution.

  • AWS SDK v1 dependencies are shaded: AWS SDK v1 dependencies bundled with the SAP Databricks runtime are now shaded and no longer directly available on the classpath. If your code depends on AWS SDK v1 libraries previously provided by the SAP Databricks runtime, add them as explicit dependencies in your project. This change prepares for the migration to AWS SDK v2, following the end of AWS support for SDK v1.

  • Fix incorrect EPSG authority for ESRI-defined SRID 102100: The Coordinate Reference System (CRS) mapping for SRID 102100 now correctly uses ESRI:102100 instead of the incorrect EPSG:102100. This fix ensures geospatial data is stored with the correct authority for better interoperability with other systems.

Version 18.1​

April 20, 2026

This serverless compute release roughly corresponds to SAP Databricks Runtime 18.1.

New features​

  • Schema evolution with INSERT statements: Use the WITH SCHEMA EVOLUTION clause with SQL INSERT statements to automatically evolve the target table's schema during insert operations. The clause is supported for INSERT INTO, INSERT OVERWRITE, and INSERT INTO ... REPLACE forms.

  • Preserved NULL struct values in INSERT operations: INSERT operations with schema evolution or implicit casting preserve NULL struct values when the source and target tables have differing struct field orders.

  • parse_timestamp SQL function: The parse_timestamp SQL function parses timestamp strings using multiple patterns. The function runs on the Photon engine for improved performance.

  • max_by and min_by with optional limit: The aggregate functions max_by and min_by now accept an optional third argument limit (up to 100,000), returning an array of top- or bottom-K values without window functions or CTEs.

  • Vector aggregate and scalar functions: New SQL functions operate on ARRAY<FLOAT> vectors for embedding and similarity workloads, including vector_avg, vector_sum, vector_cosine_similarity, vector_inner_product, vector_l2_distance, vector_norm, and vector_normalize.

  • SQL cursor support in compound statements: SQL scripting compound statements now support cursor processing. Use DECLARE CURSOR to define a cursor, then open, fetch, and close statements to run the query and consume rows one at a time.

  • Approximate top-k sketch functions: New functions enable building and combining approximate top-K sketches for distributed top-K aggregation: approx_top_k_accumulate, approx_top_k_combine, and approx_top_k_estimate.

  • Tuple sketch functions: New aggregate and scalar functions for tuple sketch support distinct counting and aggregation over key-summary pairs.

  • New geospatial functions: The following geospatial functions are now available:

    • st_estimatesrid: Estimates the best projected spatial reference identifier (SRID) for an input geometry.
    • st_force2d: Converts a geography or geometry to its 2D representation.
    • st_nrings: Counts the total number of rings in a polygon or multipolygon, including both exterior and interior rings.
    • st_numpoints: Counts the number of non-empty points in a geography or geometry.

  • Photon support for geospatial functions: st_difference, st_intersection, and st_union now run on the Photon engine for faster performance.

Behavior changes​

  • Observation metric errors no longer fail queries: Errors during observation metric collection no longer cause query execution failures. Previously, errors in OBSERVE clauses (such as division by zero) could block or fail the entire query. Now, the query completes successfully and the error is raised when you call observation.get.

  • DESCRIBE FLOW reserved keyword: The DESCRIBE FLOW command is now available. If you have a table named flow, use DESCRIBE schema.flow, DESCRIBE TABLE flow, or DESCRIBE `flow` with backticks.

  • SpatialSQL boolean set operations: ST_Difference, ST_Intersection, and ST_Union use a new implementation with approximately 2x faster performance. Valid input geometries always produce a result. Results are normalized for consistent output and can differ after the 15th decimal place for line-segment intersections due to different formulas and order of operations.

  • Exception types for SQLSTATE: Exception types are updated to support SQLSTATE. If your code parses exceptions by string matching or catches specific exception types, update your error handling logic.

Version 18.0​

February 27, 2026

This serverless compute release roughly corresponds to SAP Databricks Runtime 18.0.

New features​

  • SQL scripting is GA: SQL scripting is now generally available on serverless compute.
  • Shared isolation for Unity Catalog Python UDFs: Unity Catalog Python UDFs now run in shared isolation mode on serverless compute.
  • SQL window functions in metric views: You can now use SQL window functions in metric views.
  • Dynamic shuffle partition adjustment in stateless streaming queries: Serverless compute now dynamically adjusts shuffle partitions for stateless streaming queries to optimize performance.
  • Literal string coalescing everywhere: String literals are now coalesced across all SQL contexts.
  • Parameter markers everywhere: Parameter markers are now supported in all SQL contexts.
  • IDENTIFIER clause everywhere: The IDENTIFIER clause is now supported in all SQL contexts.
  • New BITMAP_AND_AGG function: A new aggregate function for bitmap AND operations.
  • New Theta sketch functions: New functions for Theta sketch approximate distinct counting.
  • New KLL Sketch function library: New functions for KLL sketch quantile estimation.
  • Apache Parquet upgraded to 1.16.0: The Apache Parquet library has been upgraded to version 1.16.0.
  • New geospatial functions: st_azimuth, st_boundary, st_closestpoint, st_geogfromewkt, st_geomfromewkt.
  • Improved spatial join performance: Spatial joins now run faster on serverless compute.
  • Improved geospatial function performance: Geospatial functions have been optimized for better performance.

Behavior changes​

  • FSCK REPAIR TABLE includes metadata repair by default.
  • Python UDF execution unified (TIMESTAMP timezone behavior change).
  • Time travel restrictions and VACUUM retention behavior updated.
  • BinaryType maps to bytes by default in PySpark.
  • Partition columns materialized in Parquet files.
  • DESCRIBE TABLE output includes metadata column.

Serverless environment version 5 is now available​

February 25, 2026

Serverless environment version 5 is now available and includes updated system libraries and security patches.

Version 17.3​

October 28, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.3 LTS.

New features​

  • LIMIT ALL support for recursive CTEs: You can now use the LIMIT ALL clause with recursive common table expressions (rCTEs) to explicitly specify that no row limit should be applied to the query results.

  • Appending to files in Unity Catalog volumes returns correct error: Attempting to append to existing files in Unity Catalog volumes now returns a more descriptive error message to help you understand and resolve the issue.

  • st_dump function support: You can now use the st_dump function to decompose a geometry object into its constituent parts, returning a set of simpler geometries.

  • Polygon interior ring functions are now supported: You can now use the following functions to work with polygon interior rings:

    • st_numinteriorrings: Get the number of inner boundaries (rings) of a polygon.
    • st_interiorringn: Extract the n-th inner boundary of a polygon and return it as a linestring.

  • EXECUTE IMMEDIATE using constant expressions: The EXECUTE IMMEDIATE statement now supports using constant expressions in the query string, allowing for more flexible dynamic SQL execution.

  • Allow spark.sql.files.maxPartitionBytes in serverless compute: You can now configure the spark.sql.files.maxPartitionBytes Spark configuration parameter on serverless compute to control the maximum number of bytes to pack into a single partition when reading files.

Behavior changes​

  • Add metadata column to DESCRIBE QUERY and DESCRIBE TABLE: The DESCRIBE QUERY and DESCRIBE TABLE commands now include a metadata column in their output, providing additional information about each column's properties and characteristics.

  • Default mode change for FSCK REPAIR TABLE command: The default mode for the FSCK REPAIR TABLE command has changed to provide more consistent behavior when repairing table metadata.

  • Correct handling of null structs when dropping NullType columns: SAP Databricks now correctly handles null struct values when dropping columns with NullType, preventing potential data corruption or unexpected behavior.

  • Improved handling of null structs in Parquet: This release includes improvements to how null struct values are handled when reading from and writing to Parquet files, ensuring more consistent and correct behavior.

  • Upgrade aws-msk-iam-auth library for Kafka: The aws-msk-iam-auth library used for Amazon MSK IAM authentication has been upgraded to the latest version, providing improved security and compatibility.

Version 17.2​

September 25, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.2.

New features​

  • ST_ExteriorRing function is now supported: You can now use the ST_ExteriorRing function to extract the outer boundary of a polygon and return it as a linestring.

  • Support TEMPORARY keyword for metric view creation: You can now use the TEMPORARY keyword when creating a metric view. Temporary metric views are visible only in the session that created them and are dropped when the session ends.

  • Use native I/O for LokiFileSystem.getFileStatus on S3: LokiFileSystem.getFileStatus now uses the native I/O stack for Amazon S3 traffic and returns org.apache.hadoop.fs.FileStatus objects instead of shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileStatus.

  • Auto Loader infers partition columns in singleVariantColumn mode: Auto Loader now infers partition columns from file paths when ingesting data as a semi-structured variant type using the singleVariantColumn option. Previously, partition columns were not automatically detected.

Behavior changes​

  • DESCRIBE CONNECTION shows environment settings for JDBC connections: SAP Databricks now includes user-defined environment settings in the DESCRIBE CONNECTION output for JDBC connections that support custom drivers and run in isolation. Other connection types remain unchanged.

  • Option to truncate uniform history during managed tables migration: You can now truncate uniform history when migrating tables with Uniform/Iceberg enabled using ALTER TABLE...SET MANAGED. This simplifies migrations and reduces downtime compared to disabling and re-enabling Uniform manually.

  • Correct results for split with empty regex and positive limit: SAP Databricks now returns correct results when using split function with an empty regex and a positive limit. Previously, the function incorrectly truncated the remaining string instead of including it in the last element.

  • Fix url_decode and try_url_decode error handling in Photon: In Photon, try_url_decode() and url_decode() with failOnError = false now return NULL for invalid URL-encoded strings instead of failing the query.

  • Shared execution environment for Unity Catalog Python UDTFs: SAP Databricks now shares the execution environment for Python user-defined table functions (UDTFs) from the same owner and Spark session. An optional STRICT ISOLATION clause is available to disable sharing for UDTFs with side effects, such as modifying environment variables or executing arbitrary code.

Version 17.1​

August 19, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.1.

New features​

  • Reduced memory usage for wide schemas in Photon writer: Enhancements were made to the Photon engine that significantly reduce memory usage for wide schemas, addressing scenarios that previously resulted in out-of-memory errors.

Behavior changes​

  • Error thrown for invalid CHECK constraints: SAP Databricks now throws an AnalysisException if a CHECK constraint expression cannot be resolved during constraint validation.

  • Pulsar connector no longer exposes Bouncy Castle: The Bouncy Castle library is now shaded in the Pulsar connector to prevent classpath conflicts. As a result, Spark jobs can no longer access org.bouncycastle.* classes from the connector. If your code depends on Bouncy Castle, install the library manually on serverless environment.

Serverless environment version 4​

August 13, 2025

Environment version 4 is now available in your serverless notebooks and jobs. This environment version includes library upgrades and API updates.

Version 17.0​

July 24, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.0.

New features​

  • SQL procedure support: SQL scripts can now be encapsulated in a procedure stored as a reusable asset in Unity Catalog. You can create a procedure using the CREATE PROCEDURE command, and then call it using the CALL command.

  • Set a default collation for SQL Functions: Using the new DEFAULT COLLATION clause in the CREATE FUNCTION command defines the default collation used for STRING parameters, the return type, and STRING literals in the function body.

  • Recursive common table expressions (rCTE) support: SAP Databricks now supports navigation of hierarchical data using recursive common table expressions (rCTEs). Use a self-referencing CTE with UNION ALL to follow the recursive relationship.

  • PySpark and Spark Connect now support the DataFrames df.mergeInto API: PySpark and Spark Connect now support the df.mergeInto API.

  • Support ALL CATALOGS in SHOW SCHEMAS: The SHOW SCHEMAS syntax is updated to accept ALL CATALOGS, allowing you to iterate through all active catalogs that support namespaces. The output attributes now include a catalog column indicating the catalog of the corresponding namespace.

  • Liquid clustering now compacts deletion vectors more efficiently: Delta tables with liquid clustering now apply physical changes from deletion vectors more efficiently when OPTIMIZE is running.

  • Allow non-deterministic expressions in UPDATE/INSERT column values for MERGE operations: SAP Databricks now allows the use of non-deterministic expressions in updated and inserted column values of MERGE operations. For example, you can now generate dynamic or random values for columns using expressions like rand().

  • Change Delta MERGE Python APIs to return DataFrame instead of Unit: The Python MERGE APIs (such as DeltaMergeBuilder) now also return a DataFrame like the SQL API does, with the same results.

On this page