Serverless environment version 5
This page outlines the system environment information for serverless environment version 5.
To ensure application compatibility, serverless workloads use a versioned API, known as the environment version, that remains compatible with newer serverless versions.
To select a base environment, use the Base environment selector in the Environment side panel in your serverless notebooks. See Select a base environment.
New features and improvements
The following new features and improvements are available in serverless environment 5.
MLflow 3.8.1 support
Serverless environment 5 includes MLflow 3.8.1, a major upgrade from MLflow 2.22.0 in environment version 4. This upgrade includes numerous new features, improvements, and bug fixes. For a complete list of changes, see the MLflow release notes.
Serverless JAR jobs
This feature is in Public Preview.
Serverless JAR jobs are now supported in environment version 5. This feature allows you to run JAR-based applications on serverless compute infrastructure.
Arrow optimization enabled by default for Python UDFs
In environment version 5, Arrow optimization is enabled by default for Python UDFs, significantly improving UDF execution performance. Additionally, the Arrow-based pandas serializer conversion has been improved for better efficiency.
This represents a behavioral change from previous versions, where Arrow optimization was opt-in.
Behavioral changes
The following behavioral changes are introduced in serverless environment 5.
BinaryType consistently maps to Python bytes
In PySpark, BinaryType now consistently maps to Python bytes. Previously, PySpark mapped BinaryType to either bytes or bytearray depending on the context.
For more information, see SPARK-53696.
Linux Kernel-level socket API for cryptography
Starting with environment version 5, the Linux Kernel-level crypto socket API that operates on non-FIPS-compliant ciphers may fail non-deterministically. Do not use this API for non-FIPS compliance crypto operations.
API updates
Serverless environment 5 includes the following API updates:
- SPARK-53635 Support Scala UDFs with input args of type Seq[Row]
- SPARK-54220 NullType/VOID/UNKNOWN Type Support in Parquet
- SPARK-54153 Support profiling iterator based Python UDFs
- SPARK-54213 Remove Python 3.9 from Spark Connect
- SPARK-53977 Support logging in UDTFs
- SPARK-53976 Support logging in Pandas/Arrow UDFs
- SPARK-53573 Allow coalescing string literals everywhere
- SPARK-54269 Upgrade cloudpickle to 3.1.2 for Python 3.14
- SPARK-54287 Add Python 3.14 support in pyspark-client and pyspark-connect
- SPARK-53614 Add Iterator[pandas.DataFrame] support to applyInPandas
- SPARK-53921 Introduce GeometryType and GeographyType to PySpark API
- SPARK-53920 Introduce GeometryType and GeographyType to Java API
- SPARK-53956 Support TIME in the try_make_timestamp function in PySpark
- SPARK-53930 Support TIME in the make_timestamp function in PySpark
- SPARK-53111 Implement the time_diff function in PySpark
- SPARK-53877 Introduce BITMAP_AND_AGG function
- SPARK-53357 Update pandas to 2.3.2
- SPARK-52980 Support Arrow Python UDTFs
- SPARK-52844 Update numpy to 1.22
- SPARK-50359 Upgrade PyArrow to 18.0
- SPARK-50564 Upgrade protobuf Python package to 5.29.1
- SPARK-50601 Support withColumns / withColumnsRenamed in subqueries
- SPARK-51814 Introduce a new API transformWithState in PySpark
- SPARK-52821 add int->DecimalType pyspark udf return type coercion
- SPARK-53112 Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark
- SPARK-53319 Support the time type by try_make_timestamp_ltz()
- SPARK-53696 Default to bytes for BinaryType in PySpark
- SPARK-55090 Implement DataFrame.toJSON in Python Client
System environment
- Operating System: Ubuntu 24.04.3 LTS
- Python: 3.12.3
- Databricks Connect: 18 (Databricks Connect is continuously updated in the latest serverless environment version. Run
pip listto confirm the exact version in your current environment.) - Scala: 2.13.16
- JDK: 21
Installed Python libraries
To reproduce serverless environment 5 in your local Python virtual environment, download the requirements-env-5.txt file and run pip install -r requirements-env-5.txt. This command installs all of the open source libraries from serverless environment 5.
Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
aiohappyeyeballs | 2.4.4 | aiohttp | 3.11.10 | aiosignal | 1.2.0 |
annotated-doc | 0.0.4 | annotated-types | 0.7.0 | anyio | 4.7.0 |
argon2-cffi | 21.3.0 | argon2-cffi-bindings | 21.2.0 | arro3-core | 0.6.5 |
arrow | 1.3.0 | asttokens | 3.0.0 | astunparse | 1.6.3 |
async-lru | 2.0.4 | attrs | 24.3.0 | autocommand | 2.2.2 |
azure-common | 1.1.28 | azure-core | 1.37.0 | azure-identity | 1.20.0 |
azure-mgmt-core | 1.6.0 | azure-mgmt-web | 8.0.0 | azure-storage-blob | 12.28.0 |
azure-storage-file-datalake | 12.22.0 | babel | 2.16.0 | backports.tarfile | 1.2.0 |
beautifulsoup4 | 4.12.3 | black | 24.10.0 | bleach | 6.2.0 |
blinker | 1.7.0 | boto3 | 1.40.45 | botocore | 1.40.45 |
cachetools | 5.5.1 | certifi | 2025.4.26 | cffi | 1.17.1 |
chardet | 4.0.0 | charset-normalizer | 3.3.2 | click | 8.1.8 |
cloudpickle | 3.0.0 | comm | 0.2.1 | contourpy | 1.3.1 |
cryptography | 44.0.1 | cycler | 0.11.0 | Cython | 3.1.5 |
databricks-agents | 1.9.1 | databricks-connect | 18.0.2 | databricks-sdk | 0.67.0 |
dataclasses-json | 0.6.7 | dbus-python | 1.3.2 | debugpy | 1.8.11 |
decorator | 5.1.1 | defusedxml | 0.7.1 | delta-spark | 3.4.0+databricks.connect.18.0.2 |
deltalake | 1.1.4 | Deprecated | 1.2.18 | distlib | 0.3.9 |
docstring-to-markdown | 0.11 | executing | 1.2.0 | facets-overview | 1.1.1 |
fastapi | 0.128.0 | fastjsonschema | 2.21.1 | filelock | 3.17.0 |
fonttools | 4.55.3 | fqdn | 1.5.1 | frozenlist | 1.5.0 |
fsspec | 2023.5.0 | gitdb | 4.0.11 | GitPython | 3.1.43 |
google-api-core | 2.28.1 | google-auth | 2.47.0 | google-cloud-core | 2.5.0 |
google-cloud-storage | 3.7.0 | google-crc32c | 1.8.0 | google-resumable-media | 2.8.0 |
googleapis-common-protos | 1.65.0 | grpcio | 1.67.0 | grpcio-status | 1.67.0 |
h11 | 0.16.0 | hf-xet | 1.2.0 | httpcore | 1.0.9 |
httplib2 | 0.20.4 | httpx | 0.28.1 | huggingface_hub | 1.2.4 |
idna | 3.7 | importlib_metadata | 8.5.0 | inflect | 7.3.1 |
iniconfig | 1.1.1 | ipyflow-core | 0.0.209 | ipykernel | 6.29.5 |
ipython | 8.30.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.8.1 |
isodate | 0.7.2 | isoduration | 20.11.0 | jaraco.collections | 5.1.0 |
jaraco.context | 5.3.0 | jaraco.functools | 4.0.1 | jaraco.text | 3.12.1 |
jedi | 0.19.2 | Jinja2 | 3.1.6 | jiter | 0.12.0 |
jmespath | 1.0.1 | joblib | 1.4.2 | json5 | 0.9.25 |
jsonpatch | 1.33 | jsonpointer | 3.0.0 | jsonschema | 4.23.0 |
jsonschema-specifications | 2023.7.1 | jupyter-events | 0.12.0 | jupyter-lsp | 2.2.5 |
jupyter_client | 8.6.3 | jupyter_core | 5.7.2 | jupyter_server | 2.15.0 |
jupyter_server_terminals | 0.5.3 | jupyterlab | 4.3.4 | jupyterlab_pygments | 0.3.0 |
jupyterlab_server | 2.27.3 | jupyterlab_widgets | 1.1.11 | kiwisolver | 1.4.8 |
langchain-core | 1.2.6 | langchain-openai | 1.1.6 | langsmith | 0.6.1 |
launchpadlib | 1.11.0 | lazr.restfulclient | 0.14.6 | lazr.uri | 1.0.6 |
litellm | 1.75.9 | markdown-it-py | 2.2.0 | MarkupSafe | 3.0.2 |
marshmallow | 3.26.2 | matplotlib | 3.10.0 | matplotlib-inline | 0.1.7 |
mccabe | 0.7.0 | mdurl | 0.1.0 | mistune | 3.1.2 |
mlflow-skinny | 3.8.1 | mmh3 | 5.2.0 | more-itertools | 10.3.0 |
msal | 1.34.0 | msal-extensions | 1.3.1 | multidict | 6.1.0 |
mypy-extensions | 1.0.0 | nbclient | 0.10.2 | nbconvert | 7.16.6 |
nbformat | 5.10.4 | nest-asyncio | 1.6.0 | nodeenv | 1.10.0 |
notebook | 7.3.2 | notebook_shim | 0.2.4 | numpy | 2.1.3 |
oauthlib | 3.2.2 | openai | 2.14.0 | opentelemetry-api | 1.39.1 |
opentelemetry-proto | 1.39.1 | opentelemetry-sdk | 1.39.1 | opentelemetry-semantic-conventions | 0.60b1 |
orjson | 3.11.5 | overrides | 7.4.0 | packaging | 24.2 |
pandas | 2.2.3 | pandocfilters | 1.5.0 | parso | 0.8.4 |
pathspec | 0.10.3 | patsy | 1.0.1 | pexpect | 4.8.0 |
pillow | 11.1.0 | pip | 25.0.1 | platformdirs | 4.3.7 |
plotly | 5.24.1 | pluggy | 1.5.0 | prometheus_client | 0.21.1 |
prompt-toolkit | 3.0.43 | propcache | 0.3.1 | proto-plus | 1.27.0 |
protobuf | 5.29.4 | psutil | 5.9.0 | psycopg2 | 2.9.11 |
ptyprocess | 0.7.0 | pure-eval | 0.2.2 | py4j | 0.10.9.9 |
pyarrow | 21.0.0 | pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 |
pyccolo | 0.0.71 | pycparser | 2.21 | pydantic | 2.10.6 |
pydantic_core | 2.27.2 | pyflakes | 3.2.0 | Pygments | 2.19.1 |
PyGObject | 3.48.2 | pyiceberg | 0.10.0 | PyJWT | 2.10.1 |
pyodbc | 5.2.0 | pyparsing | 3.2.0 | pyright | 1.1.394 |
pyroaring | 1.0.3 | pyspark | 4.1.0+databricks.connect.18.0.2 | pytest | 8.3.5 |
python-dateutil | 2.9.0.post0 | python-dotenv | 1.2.1 | python-json-logger | 3.2.1 |
python-lsp-jsonrpc | 1.1.2 | python-lsp-server | 1.12.2 | pytoolconfig | 1.2.6 |
pytz | 2024.1 | PyYAML | 6.0.2 | pyzmq | 26.2.0 |
referencing | 0.30.2 | regex | 2024.11.6 | requests | 2.32.3 |
requests-toolbelt | 1.0.0 | rfc3339-validator | 0.1.4 | rfc3986-validator | 0.1.1 |
rich | 13.9.4 | rope | 1.13.0 | rpds-py | 0.22.3 |
rsa | 4.9.1 | s3transfer | 0.14.0 | scikit-learn | 1.6.1 |
scipy | 1.15.3 | seaborn | 0.13.2 | Send2Trash | 1.8.2 |
setuptools | 78.1.1 | shellingham | 1.5.4 | six | 1.17.0 |
smmap | 5.0.0 | sniffio | 1.3.0 | sortedcontainers | 2.4.0 |
soupsieve | 2.5 | sqlparse | 0.5.5 | ssh-import-id | 5.11 |
stack-data | 0.6.3 | starlette | 0.50.0 | strictyaml | 1.7.3 |
tenacity | 9.0.0 | terminado | 0.17.1 | threadpoolctl | 3.5.0 |
tiktoken | 0.12.0 | tinycss2 | 1.4.0 | tokenize_rt | 6.1.0 |
tokenizers | 0.22.2 | tomli | 2.0.1 | tornado | 6.5.1 |
tqdm | 4.67.1 | traitlets | 5.14.3 | typeguard | 4.3.0 |
typer-slim | 0.21.1 | types-python-dateutil | 2.9.0.20251115 | typing-inspect | 0.9.0 |
typing_extensions | 4.12.2 | tzdata | 2024.1 | ujson | 5.10.0 |
unattended-upgrades | 0.1 | uri-template | 1.3.0 | urllib3 | 2.3.0 |
uuid_utils | 0.12.0 | uvicorn | 0.40.0 | virtualenv | 20.29.3 |
wadllib | 1.3.6 | wcwidth | 0.2.5 | webcolors | 25.10.0 |
webencodings | 0.5.1 | websocket-client | 1.8.0 | whatthepatch | 1.0.2 |
wheel | 0.45.1 | whenever | 0.7.3 | widgetsnbextension | 3.6.6 |
wrapt | 1.17.0 | yapf | 0.40.2 | yarl | 1.18.0 |
zipp | 3.21.0 | zstandard | 0.23.0 |
Installed Java and Scala libraries (Scala 2.13 cluster version)
Group ID | Artifact ID | Version |
|---|---|---|
com.databricks | databricks-connect_2.13 | 18.0.0 |
com.fasterxml.jackson.core | jackson-annotations | 2.15.2 |
com.fasterxml.jackson.core | jackson-core | 2.15.2 |
com.fasterxml.jackson.core | jackson-databind | 2.15.2 |
com.lihaoyi | ammonite-compiler-interface_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-compiler_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-interp-api_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-interp_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-repl-api_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-repl_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-runtime_2.13.16 | 3.0.2 |
com.lihaoyi | ammonite-util_2.13 | 3.0.2 |
com.lihaoyi | fansi_2.13 | 0.5.0 |
com.lihaoyi | os-lib_2.13 | 0.11.3 |
com.lihaoyi | pprint_2.13 | 0.9.0 |
com.lihaoyi | scalaparse_2.13 | 3.1.1 |
org.apache.logging.log4j | log4j-api | 2.20.0 |
org.apache.logging.log4j | log4j-core | 2.20.0 |
org.json4s | json4s-core_2.13 | 4.0.7 |
org.json4s | json4s-jackson_2.13 | 4.0.7 |
sh.almond | channels_2.13 | 0.14.1-1 |
sh.almond | interpreter-api_2.13 | 0.14.1-1 |
sh.almond | interpreter_2.13 | 0.14.1-1 |
sh.almond | jupyter-api_2.13 | 0.14.1-1 |
sh.almond | kernel_2.13 | 0.14.1-1 |
sh.almond | logger_2.13 | 0.14.1-1 |
sh.almond | protocol_2.13 | 0.14.1-1 |
sh.almond | scala-interpreter_2.13.16 | 0.14.1-1 |
sh.almond | scala-kernel_2.13.16 | 0.14.1-1 |
sh.almond | shared-directives_2.13 | 0.14.1-1 |