Serverless environment version 3

This article outlines the system environment information for serverless environment version 3.

To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.

You can select the environment version using the Environment side panel in your serverless notebooks. See Select an environment version.

New features and improvements

The following new features and improvements are available in serverless environment 3.

API updates

June 13, 2025

Serverless environment 3 includes the following API updates:

• Added filter pushdown API support to Python data sources. See Add filter pushdown API support to Python data sources.
• Python UDF traceback now includes frames from both the driver and executor, along with client frames. See Python UDF traceback improvement.
• New listagg and string_agg functions aggregate STRING and BINARY values in a group. See New listagg and string_agg functions.
• variant_get and get_json_object now consider leading spaces in paths in Apache Spark. See variant_get and get_json_object now consider leading spaces in paths in Apache Spark.
• SPARK-51079 Support large variable types in pandas UDF, createDataFrame, and toPandas with Arrow.
• SPARK-51186 Add StreamingPythonRunnerInitializationException to PySpark base exception.
• SPARK-51112 Avoid using pyarrow's to_pandas on an empty table.
• SPARK-51506 [PYTHON][ss] Do not enforce users to implement close() in TransformWithStateInPandas
• SPARK-51425 [Connect] Add client API to set custom operation_id
• SPARK-51206 [PYTHON][connect] Move Arrow conversion helpers out of Spark Connect

Databricks Connect upgraded to 16.3

June 13, 2025

Use the features and improvements available on Databricks Connect for Databricks Runtime 16.3. See Databricks Connect for Databricks Runtime 16.3.

Improved Python syntax error highlighting

June 13, 2025

Python syntax error highlighting will see the following improvements:

• Faster error handling latency.
• Support for Python type error highlighting.
• Linter configurability through pyproject.toml files.

See Python error highlighting.

Git CLI support in web terminal and notebook

June 13, 2025

You can now use the Git CLI in a serverless notebook and the serverless notebook's web terminal.

Behavioral change regarding conflicting environment versions

June 13, 2025

In cases where a serverless environment version is declared in both the notebook's Environment panel and in a custom base environment file, the base environment's version takes precedence unless both declared versions are under version 3, in which case the notebook's version is used.

For example:

• If the notebook uses v1 and the base environment uses v3, the workload will use v3.
• If the notebook uses v1 and the base environment uses v2, the workload will use v1.
• If the notebook uses v3 and the base environment uses v1, the workload will use v1.
• If the notebook uses v2 and the base environment uses v1, the workload will use v2.

System environment

• Operating System: Ubuntu 24.04.2 LTS
• Python: 3.12.3
• Databricks Connect: 16.3.2

Installed Python libraries

Library	Version	Library	Version	Library	Version
annotated-types	0.7.0	anyio	4.2.0	asttokens	2.0.5
astunparse	1.6.3	autocommand	2.2.2	azure-core	1.33.0
azure-storage-blob	12.23.0	azure-storage-file-datalake	12.17.0	backports.tarfile	1.2.0
black	24.4.2	blinker	1.7.0	boto3	1.34.69
botocore	1.34.69	cachetools	5.3.3	certifi	2024.6.2
cffi	1.16.0	chardet	4.0.0	charset-normalizer	2.0.4
click	8.1.7	cloudpickle	3.0.0	comm	0.2.1
contourpy	1.2.0	cryptography	42.0.5	cycler	0.11.0
Cython	3.0.11	databricks-connect	16.3.2	databricks-sdk	0.49.0
dbus-python	1.3.2	debugpy	1.6.7	decorator	5.1.1
Deprecated	1.2.18	dill	0.3.8	distlib	0.3.8
executing	0.8.3	facets-overview	1.1.1	fastapi	0.115.12
filelock	3.15.4	fonttools	4.51.0	gitdb	4.0.11
GitPython	3.1.37	google-api-core	2.20.0	google-auth	2.38.0
google-cloud-core	2.4.3	google-cloud-storage	3.1.0	google-crc32c	1.7.1
google-resumable-media	2.7.2	googleapis-common-protos	1.69.2	grpcio	1.71.0
grpcio-status	1.71.0	h11	0.14.0	httplib2	0.20.4
idna	3.7	importlib-metadata	7.0.1	inflect	7.3.1
iniconfig	1.1.1	ipyflow-core	0.0.206	ipykernel	6.29.5
ipython	8.32.0	ipython-genutils	0.2.0	ipywidgets	7.7.2
isodate	0.7.2	jaraco.collections	5.1.0	jaraco.context	5.3.0
jaraco.functools	4.0.1	jaraco.text	3.12.1	jedi	0.19.1
jmespath	1.0.1	joblib	1.4.2	jupyter_client	8.6.0
jupyter_core	5.7.2	kiwisolver	1.4.4	launchpadlib	1.11.0
lazr.restfulclient	0.14.6	lazr.uri	1.0.6	matplotlib	3.8.4
matplotlib-inline	0.1.6	mlflow-skinny	2.21.3	more-itertools	10.3.0
mypy-extensions	1.0.0	nest-asyncio	1.6.0	numpy	1.26.4
oauthlib	3.2.2	opentelemetry-api	1.31.1	opentelemetry-sdk	1.31.1
opentelemetry-semantic-conventions	0.52b1	packaging	24.1	pandas	1.5.3
parso	0.8.3	pathspec	0.10.3	patsy	0.5.6
pexpect	4.8.0	pillow	10.3.0	pip	25.0.1
platformdirs	3.10.0	plotly	5.22.0	pluggy	1.5.0
prompt-toolkit	3.0.43	proto-plus	1.26.1	protobuf	5.29.4
psutil	5.9.0	psycopg2	2.9.3	ptyprocess	0.7.0
pure-eval	0.2.2	py4j	0.10.9.7	pyarrow	15.0.2
pyasn1	0.4.8	pyasn1-modules	0.2.8	pyccolo	0.0.68
pycparser	2.21	pydantic	2.8.2	pydantic_core	2.20.1
Pygments	2.15.1	PyGObject	3.48.2	PyJWT	2.7.0
pyodbc	5.0.1	pyparsing	3.0.9	pytest	8.3.0
python-dateutil	2.9.0.post0	python-lsp-jsonrpc	1.1.2	pytz	2024.1
PyYAML	6.0.1	pyzmq	25.1.2	requests	2.32.2
rsa	4.9	s3transfer	0.10.4	scikit-learn	1.4.2
scipy	1.13.1	seaborn	0.13.2	setuptools	75.8.0
six	1.16.0	smmap	5.0.0	sniffio	1.3.0
sqlparse	0.5.3	ssh-import-id	5.11	stack-data	0.2.0
starlette	0.46.1	statsmodels	0.14.2	tenacity	8.2.2
threadpoolctl	2.2.0	tokenize-rt	4.2.1	tomli	2.0.1
tornado	6.4.1	traitlets	5.14.3	typeguard	4.3.0
typing_extensions	4.11.0	tzdata	2024.1	ujson	5.10.0
unattended-upgrades	0.1	urllib3	2.2.2	uvicorn	0.34.0
virtualenv	20.29.2	wadllib	1.3.6	wcwidth	0.2.5
wheel	0.45.1	wrapt	1.14.1	zipp	3.17.0
zstandard	0.23.0