Serverless environment version 2
This article outlines the system environment information for serverless environment version 2.
To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.
You can select the environment version using the Environment side panel in your serverless notebooks. See Configure the serverless environment.
For more on environment versions, see Serverless compute release notes.
New features and improvements
The following new features and improvements are available in serverless environment 2.
Web terminal enabled on serverless compute
April 3, 2025
The web terminal is now enabled on serverless environment version 2. For more information on how to use the web terminal, see Run shell commands in Databricks web terminal.
The VARIANT data type can no longer be used with operations that require comparisons
February 5, 2025
You cannot use the following clauses or operators in queries that include a VARIANT data type:
DISTINCTINTERSECTEXCEPTUNIONDISTRIBUTE BY
Additionally, you cannot use these DataFrame functions:
df.dropDuplicates()df.repartition()
These operations perform comparisons, and comparisons that use the VARIANT data type produce undefined results and are not supported in Databricks. If you use the VARIANT type in your Databricks workloads or tables, Databricks recommends the following changes:
- Update queries or expressions to explicitly cast
VARIANTvalues to non-VARIANTdata types. - If you have fields that must be used with any of the above operations, extract those fields from the
VARIANTdata type and store them using non-VARIANTdata types.
See Query variant data.
Notebooks are supported as workspace files
January 23, 2025
Notebooks are supported as workspace files in serverless environment 2. You can programmatically write, read, and delete notebooks just as you would any other file. This allows for programmatic interaction with notebooks from anywhere the workspace filesystem is available. For more information, see Notebooks as workspace files.
Task progress bar added to serverless compute
December 16, 2024
A new task progress bar has been added to notebook cells running on serverless compute environment version 2. This progress bar indicates the execution progress of the cell's Spark code.
System environment
- Operating System: Ubuntu 22.04.4 LTS
- Python: 3.11.10
- Databricks Connect: 15.4.5
Installed Python libraries
Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
asttokens | 2.0.5 | astunparse | 1.6.3 | autocommand | 2.2.2 |
azure-core | 1.31.0 | azure-storage-blob | 12.19.1 | azure-storage-file-datalake | 12.14.0 |
backports.tarfile | 1.2.0 | black | 23.3.0 | blinker | 1.4 |
boto3 | 1.34.39 | botocore | 1.34.39 | cachetools | 5.5.0 |
certifi | 2023.7.22 | cffi | 1.15.1 | chardet | 4.0.0 |
charset-normalizer | 2.0.4 | click | 8.0.4 | cloudpickle | 3.0.0 |
comm | 0.1.2 | contourpy | 1.0.5 | cryptography | 41.0.3 |
cycler | 0.11.0 | Cython | 0.29.32 | databricks-connect | 15.4.5 |
databricks-sdk | 0.36.0 | dbus-python | 1.2.18 | debugpy | 1.6.7 |
decorator | 5.1.1 | dill | 0.3.6 | distlib | 0.3.9 |
entrypoints | 0.4 | executing | 0.8.3 | facets-overview | 1.1.1 |
filelock | 3.13.4 | fonttools | 4.25.0 | gitdb | 4.0.11 |
GitPython | 3.1.43 | google-api-core | 2.18.0 | google-auth | 2.35.0 |
google-cloud-core | 2.4.1 | google-cloud-storage | 2.18.2 | google-crc32c | 1.6.0 |
google-resumable-media | 2.7.2 | googleapis-common-protos | 1.65.0 | grpcio | 1.67.0 |
grpcio-status | 1.67.0 | httplib2 | 0.20.2 | idna | 3.4 |
importlib-metadata | 6.0.0 | importlib_resources | 6.4.0 | inflect | 7.3.1 |
ipyflow-core | 0.0.201 | ipykernel | 6.28.0 | ipython | 8.25.0 |
ipython-genutils | 0.2.0 | ipywidgets | 7.7.2 | isodate | 0.7.2 |
jaraco.collections | 5.1.0 | jaraco.context | 5.3.0 | jaraco.functools | 4.0.1 |
jaraco.text | 3.12.1 | jedi | 0.18.1 | jeepney | 0.7.1 |
jmespath | 0.10.0 | joblib | 1.2.0 | jupyter_client | 7.4.9 |
jupyter_core | 5.3.0 | keyring | 23.5.0 | kiwisolver | 1.4.4 |
launchpadlib | 1.10.16 | lazr.restfulclient | 0.14.4 | lazr.uri | 1.0.6 |
matplotlib | 3.7.2 | matplotlib-inline | 0.1.6 | mlflow-skinny | 2.11.4 |
more-itertools | 8.10.0 | mypy-extensions | 0.4.3 | nest-asyncio | 1.5.6 |
numpy | 1.23.5 | oauthlib | 3.2.0 | packaging | 23.2 |
pandas | 1.5.3 | parso | 0.8.3 | pathspec | 0.10.3 |
patsy | 0.5.3 | pexpect | 4.8.0 | pillow | 10.3.0 |
pip | 24.2 | platformdirs | 3.10.0 | plotly | 5.9.0 |
prompt_toolkit | 3.0.48 | proto-plus | 1.25.0 | protobuf | 5.28.3 |
psutil | 5.9.0 | psycopg2 | 2.9.3 | ptyprocess | 0.7.0 |
pure-eval | 0.2.2 | py4j | 0.10.9.7 | pyarrow | 14.0.1 |
pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 | pyccolo | 0.0.65 |
pycparser | 2.21 | pydantic | 1.10.6 | Pygments | 2.15.1 |
PyGObject | 3.42.1 | PyJWT | 2.3.0 | pyodbc | 4.0.39 |
pyparsing | 3.0.9 | python-dateutil | 2.8.2 | python-lsp-jsonrpc | 1.1.2 |
pytz | 2022.7 | PyYAML | 6.0 | pyzmq | 25.1.2 |
requests | 2.31.0 | rsa | 4.9 | s3transfer | 0.10.3 |
scikit-learn | 1.3.0 | scipy | 1.11.1 | seaborn | 0.12.2 |
SecretStorage | 3.3.1 | setuptools | 75.1.0 | six | 1.16.0 |
smmap | 5.0.1 | sqlparse | 0.5.1 | ssh-import-id | 5.11 |
stack-data | 0.2.0 | statsmodels | 0.14.0 | tenacity | 8.2.2 |
threadpoolctl | 2.2.0 | tokenize-rt | 4.2.1 | tomli | 2.0.1 |
tornado | 6.3.2 | traitlets | 5.13.0 | typeguard | 4.3.0 |
typing_extensions | 4.10.0 | tzdata | 2022.1 | ujson | 5.4.0 |
unattended-upgrades | 0.1 | urllib3 | 1.26.16 | virtualenv | 20.26.6 |
wadllib | 1.3.6 | wcwidth | 0.2.5 | wheel | 0.38.4 |
zipp | 3.11.0 | zstandard | 0.23.0 |