Pular para o conteúdo principal

Serverless environment version 4

This article outlines the system environment information for serverless environment version 4.

To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.

You can select the environment version using the Environment side panel in your serverless notebooks. See Select an environment version.

New features and improvements​

The following new features and improvements are available in serverless environment 4.

Databricks Connect upgraded to 17.0​

Use the features and improvements available on Databricks Connect for Databricks Runtime 17.0.

Scalar Python UDFs now support service credentials​

Scalar Python UDFs can use Unity Catalog service credentials to securely access external cloud services. For more information, see Service credentials in Scalar Python UDFs.

PySpark and Spark Connect now support the DataFrames df.mergeInto API​

PySpark and Spark Connect now support the df.mergeInto API, which was previously only available for Scala.

API updates​

Serverless environment 4 includes the following API updates:

  • SPARK-50915 Add getCondition and deprecate getErrorClass in PySparkException
  • SPARK-50719 Support interruptOperation for PySpark
  • SPARK-50718 Support addArtifact(s) for PySpark
  • SPARK-49530 Support pie subplots in PySpark plotting
  • SPARK-50357 Support Interrupt(Tag|All) APIs for PySpark
  • SPARK-51178 Raise proper PySpark error instead of SparkConnectGrpcException
  • SPARK-51227 Fix PySpark Connect _minimum_grpc_version to 1.67.0
  • SPARK-50778 Add metadataColumn to PySpark DataFrame
  • SPARK-50311 Support (add|remove|get|clear)Tag(s) APIs for PySpark
  • SPARK-50310 Add a flag to disable DataFrameQueryContext for PySpark
  • SPARK-50238 Add Variant Support in PySpark UDFs/UDTFs/UDAFs and Python UC UDFs
  • SPARK-50183 Unify the internal functions for Pandas API and PySpark Plotting
  • SPARK-50170 Move _invoke_internal_function_over_columns to pyspark.sql.utils
  • SPARK-50167 Improve PySpark plotting error messages and imports
  • SPARK-48961 Make the parameter naming of PySparkException consistent with JVM
  • SPARK-49567 Use classic instead of vanilla from PySpark code base
  • SPARK-48755 transformWithState PySpark base implementation and ValueState support
  • SPARK-48714 Implement DataFrame.mergeInto in PySpark
  • SPARK-47365 Add toArrow() DataFrame method to PySpark
  • SPARK-48075 Type checking for PySpark avro functions
  • SPARK-46984 Remove pyspark.copy_func
  • SPARK-46213 Introduce PySparkImportError for error framework
  • SPARK-46226 Migrate all remaining RuntimeError into PySpark error framework
  • SPARK-45450 Fix imports according to PEP8: pyspark.pandas and pyspark (core)

System environment​

  • Operating System: Ubuntu 24.04.2 LTS
  • Python: 3.12.3
  • Databricks Connect: 17.0.1

Installed Python libraries​

To reproduce serverless environment 4 in your local Python virtual environment, download the requirements-env-4.txt file and run pip install -r requirements-env-4.txt. This command installs all of the open source libraries from serverless environment 4.

Library

Version

Library

Version

Library

Version

annotated-types

0.7.0

anyio

4.6.2

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

arrow

1.3.0

asttokens

2.0.5

astunparse

1.6.3

async-lru

2.0.4

attrs

24.3.0

autocommand

2.2.2

azure-common

1.1.28

azure-core

1.34.0

azure-identity

1.20.0

azure-mgmt-core

1.5.0

azure-mgmt-web

8.0.0

azure-storage-blob

12.23.0

azure-storage-file-datalake

12.17.0

babel

2.16.0

backports.tarfile

1.2.0

beautifulsoup4

4.12.3

black

24.10.0

bleach

6.2.0

blinker

1.7.0

boto3

1.36.2

botocore

1.36.3

cachetools

5.5.1

certifi

2025.1.31

cffi

1.17.1

chardet

4.0.0

charset-normalizer

3.3.2

click

8.1.7

cloudpickle

3.0.0

comm

0.2.1

contourpy

1.3.1

cryptography

43.0.3

cycler

0.11.0

Cython

3.0.12

databricks-connect

17.0.1

databricks-sdk

0.49.0

dbus-python

1.3.2

debugpy

1.8.11

decorator

5.1.1

defusedxml

0.7.1

Deprecated

1.2.13

distlib

0.3.9

docstring-to-markdown

0.11

executing

0.8.3

facets-overview

1.1.1

fastapi

0.115.12

fastjsonschema

2.21.1

filelock

3.18.0

fonttools

4.55.3

fqdn

1.5.1

fsspec

2023.5.0

gitdb

4.0.11

GitPython

3.1.43

google-api-core

2.20.0

google-auth

2.40.0

google-cloud-core

2.4.3

google-cloud-storage

3.1.0

google-crc32c

1.7.1

google-resumable-media

2.7.2

googleapis-common-protos

1.65.0

grpcio

1.67.0

grpcio-status

1.67.0

h11

0.14.0

httpcore

1.0.2

httplib2

0.20.4

httpx

0.27.0

idna

3.7

importlib-metadata

6.6.0

importlib_resources

6.4.0

inflect

7.3.1

iniconfig

1.1.1

ipyflow-core

0.0.209

ipykernel

6.29.5

ipython

8.30.0

ipython-genutils

0.2.0

ipywidgets

7.8.1

isodate

0.6.1

isoduration

20.11.0

jaraco.context

5.3.0

jaraco.functools

4.0.1

jaraco.text

3.12.1

jedi

0.19.2

Jinja2

3.1.5

jmespath

1.0.1

joblib

1.4.2

json5

0.9.25

jsonpointer

3.0.0

jsonschema

4.23.0

jsonschema-specifications

2023.7.1

jupyter-events

0.10.0

jupyter-lsp

2.2.0

jupyter_client

8.6.3

jupyter_core

5.7.2

jupyter_server

2.14.1

jupyter_server_terminals

0.4.4

jupyterlab

4.3.4

jupyterlab-pygments

0.1.2

jupyterlab-widgets

1.0.0

jupyterlab_server

2.27.3

kiwisolver

1.4.8

launchpadlib

1.11.0

lazr.restfulclient

0.14.6

lazr.uri

1.0.6

markdown-it-py

2.2.0

MarkupSafe

3.0.2

matplotlib

3.10.0

matplotlib-inline

0.1.7

mccabe

0.7.0

mdurl

0.1.0

mistune

2.0.4

mlflow-skinny

2.22.0

mmh3

5.1.0

more-itertools

10.3.0

msal

1.32.3

msal-extensions

1.3.1

mypy-extensions

1.0.0

nbclient

0.8.0

nbconvert

7.16.4

nbformat

5.10.4

nest-asyncio

1.6.0

nodeenv

1.9.1

notebook

7.3.2

notebook_shim

0.2.3

numpy

2.1.3

oauthlib

3.2.2

opentelemetry-api

1.32.1

opentelemetry-sdk

1.32.1

opentelemetry-semantic-conventions

0.53b1

overrides

7.4.0

packaging

24.1

pandas

2.2.3

pandocfilters

1.5.0

parso

0.8.4

pathspec

0.10.3

patsy

1.0.1

pexpect

4.8.0

pillow

11.1.0

pip

25.0.1

platformdirs

3.10.0

plotly

5.24.1

pluggy

1.5.0

prometheus_client

0.21.0

prompt-toolkit

3.0.43

proto-plus

1.26.1

protobuf

5.29.4

psutil

5.9.0

psycopg2

2.9.3

ptyprocess

0.7.0

pure-eval

0.2.2

py4j

0.10.9.9

pyarrow

19.0.1

pyasn1

0.4.8

pyasn1-modules

0.2.8

pyccolo

0.0.71

pycparser

2.21

pydantic

2.10.6

pydantic_core

2.27.2

pyflakes

3.2.0

Pygments

2.15.1

PyGObject

3.48.2

pyiceberg

0.9.0

PyJWT

2.10.1

pyodbc

5.2.0

pyparsing

3.2.0

pyright

1.1.394

pyspark

4.0.0+databricks.connect.17.0.1

pytest

8.3.5

python-dateutil

2.9.0.post0

python-json-logger

3.2.1

python-lsp-jsonrpc

1.1.2

python-lsp-server

1.12.0

pytoolconfig

1.2.6

pytz

2024.1

PyYAML

6.0.2

pyzmq

26.2.0

referencing

0.30.2

requests

2.32.3

rfc3339-validator

0.1.4

rfc3986-validator

0.1.1

rich

13.9.4

rope

1.12.0

rpds-py

0.22.3

rsa

4.9.1

s3transfer

0.11.3

scikit-learn

1.6.1

scipy

1.15.1

seaborn

0.13.2

Send2Trash

1.8.2

setuptools

74.0.0

six

1.16.0

smmap

5.0.0

sniffio

1.3.0

sortedcontainers

2.4.0

soupsieve

2.5

sqlparse

0.5.3

ssh-import-id

5.11

stack-data

0.2.0

starlette

0.46.2

statsmodels

0.14.4

strictyaml

1.7.3

tenacity

9.0.0

terminado

0.17.1

threadpoolctl

3.5.0

tinycss2

1.4.0

tokenize_rt

6.1.0

tomli

2.0.1

tornado

6.4.2

traitlets

5.14.3

typeguard

4.3.0

types-python-dateutil

2.9.0.20241206

typing_extensions

4.12.2

tzdata

2024.1

ujson

5.10.0

unattended-upgrades

0.1

uri-template

1.3.0

urllib3

2.3.0

uvicorn

0.34.2

virtualenv

20.29.3

wadllib

1.3.6

wcwidth

0.2.5

webcolors

24.11.1

webencodings

0.5.1

websocket-client

1.8.0

whatthepatch

1.0.2

wheel

0.45.1

widgetsnbextension

3.6.6

wrapt

1.17.0

yapf

0.40.2

zipp

3.21.0

zstandard

0.23.0

Última atualização em