Environment version 3
This article outlines the system environment information for environment version 3.
To ensure compatibility for the application, serverless workloads use a versioned API, known as the environment version, which remains compatible with newer server versions.
You can select a base environment that includes this environment version using the Environment side panel in your serverless notebooks. See Select a base environment.
New features and improvements
The following new features and improvements are available in environment version 3.
API updates
Aug 18, 2025
Scalar Python UDFs now support service credentials
Scalar Python UDFs can use Unity Catalog service credentials to securely access external cloud services. For more information, see Service credentials in Python UDFs.
June 13, 2025
Environment version 3 includes the following API updates:
- Added filter pushdown API support to Python data sources. See Add filter pushdown API support to Python data sources.
- Python UDF traceback now includes frames from both the driver and executor, along with client frames. See Python UDF traceback improvement.
- New
listaggandstring_aggfunctions aggregateSTRINGandBINARYvalues in a group. See Newlistaggandstring_aggfunctions. variant_getandget_json_objectnow consider leading spaces in paths in Apache Spark. See variant_get and get_json_object now consider leading spaces in paths in Apache Spark.- SPARK-51079 Support large variable types in pandas UDF, createDataFrame, and toPandas with Arrow.
- SPARK-51186 Add
StreamingPythonRunnerInitializationExceptionto PySpark base exception. - SPARK-51112 Avoid using pyarrow's
to_pandason an empty table. - SPARK-51506 [PYTHON][ss] Do not enforce users to implement close() in
TransformWithStateInPandas - SPARK-51425 [Connect] Add client API to set custom
operation_id - SPARK-51206 [PYTHON][connect] Move Arrow conversion helpers out of Spark Connect
Databricks Connect upgraded to 16.3
June 13, 2025
Use the features and improvements available on Databricks Connect for Databricks Runtime 16.3. See Databricks Connect for Databricks Runtime 16.3.
Improved Python syntax error highlighting
June 13, 2025
Python syntax error highlighting will see the following improvements:
- Faster error handling latency.
- Support for Python type error highlighting.
- Linter configurability through
pyproject.tomlfiles.
See Python error highlighting.
Git CLI support in web terminal and notebook
June 13, 2025
You can now use the Git CLI in a serverless notebook and the serverless notebook's web terminal.
Behavioral change regarding conflicting environment versions
June 13, 2025
In cases where an environment version is declared in both the notebook's base environment selection and in a custom base environment file, the base environment file's version takes precedence unless both declared versions are under version 3, in which case the notebook's version is used.
For example:
- If the notebook uses v1 and the base environment uses v3, the workload will use v3.
- If the notebook uses v1 and the base environment uses v2, the workload will use v1.
- If the notebook uses v3 and the base environment uses v1, the workload will use v1.
- If the notebook uses v2 and the base environment uses v1, the workload will use v2.
System environment
- Operating System: Ubuntu 24.04.2 LTS
- Python: 3.12.3
- Databricks Connect: 16.4.19
Installed Python libraries
To reproduce environment version 3 in your local Python virtual environment, download the requirements-env-3.txt file and run pip install -r requirements-env-3.txt. This command installs all of the libraries from environment version 3.
Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
annotated-types | 0.7.0 | anyio | 4.2.0 | asttokens | 2.0.5 |
astunparse | 1.6.3 | azure-core | 1.33.0 | azure-storage-blob | 12.23.0 |
azure-storage-file-datalake | 12.17.0 | black | 24.4.2 | boto3 | 1.34.69 |
botocore | 1.34.69 | cachetools | 5.3.3 | certifi | 2024.6.2 |
cffi | 1.16.0 | chardet | 4.0.0 | charset-normalizer | 2.0.4 |
click | 8.1.7 | cloudpickle | 3.0.0 | comm | 0.2.1 |
contourpy | 1.2.0 | cryptography | 42.0.5 | cycler | 0.11.0 |
Cython | 3.0.11 | databricks-connect | 16.4.19 | databricks-sdk | 0.49.0 |
debugpy | 1.6.7 | decorator | 5.1.1 | Deprecated | 1.2.18 |
dill | 0.3.8 | executing | 0.8.3 | facets-overview | 1.1.1 |
fastapi | 0.115.12 | fonttools | 4.51.0 | gitdb | 4.0.11 |
GitPython | 3.1.37 | google-api-core | 2.20.0 | google-auth | 2.38.0 |
google-cloud-core | 2.4.3 | google-cloud-storage | 3.1.0 | google-crc32c | 1.7.1 |
google-resumable-media | 2.7.2 | googleapis-common-protos | 1.69.2 | grpcio | 1.71.0 |
grpcio-status | 1.71.0 | h11 | 0.14.0 | idna | 3.7 |
importlib-metadata | 7.0.1 | iniconfig | 1.1.1 | ipyflow-core | 0.0.206 |
ipykernel | 6.29.5 | ipython | 8.32.0 | ipython-genutils | 0.2.0 |
ipywidgets | 7.7.2 | isodate | 0.7.2 | jedi | 0.19.1 |
jmespath | 1.0.1 | joblib | 1.4.2 | jupyter_client | 8.6.0 |
jupyter_core | 5.7.2 | kiwisolver | 1.4.4 | matplotlib | 3.8.4 |
matplotlib-inline | 0.1.6 | mlflow-skinny | 2.21.3 | mypy-extensions | 1.0.0 |
nest-asyncio | 1.6.0 | numpy | 1.26.4 | opentelemetry-api | 1.31.1 |
opentelemetry-sdk | 1.31.1 | opentelemetry-semantic-conventions | 0.52b1 | packaging | 24.1 |
pandas | 1.5.3 | parso | 0.8.3 | pathspec | 0.10.3 |
patsy | 0.5.6 | pexpect | 4.8.0 | pillow | 10.3.0 |
pip | 25.0.1 | platformdirs | 3.10.0 | plotly | 5.22.0 |
pluggy | 1.5.0 | prompt-toolkit | 3.0.43 | proto-plus | 1.26.1 |
protobuf | 5.29.4 | psutil | 5.9.0 | psycopg2 | 2.9.3 |
ptyprocess | 0.7.0 | pure-eval | 0.2.2 | py4j | 0.10.9.9 |
pyarrow | 15.0.2 | pyasn1 | 0.4.8 | pyasn1-modules | 0.2.8 |
pyccolo | 0.0.68 | pycparser | 2.21 | pydantic | 2.8.2 |
pydantic_core | 2.20.1 | Pygments | 2.15.1 | pyodbc | 5.0.1 |
pyparsing | 3.0.9 | pytest | 8.3.0 | python-dateutil | 2.9.0.post0 |
python-lsp-jsonrpc | 1.1.2 | pytz | 2024.1 | PyYAML | 6.0.1 |
pyzmq | 25.1.2 | requests | 2.32.2 | rsa | 4.9 |
s3transfer | 0.10.4 | scikit-learn | 1.4.2 | scipy | 1.13.1 |
seaborn | 0.13.2 | smmap | 5.0.0 | sniffio | 1.3.0 |
sqlparse | 0.5.3 | stack-data | 0.2.0 | starlette | 0.46.1 |
statsmodels | 0.14.2 | tenacity | 8.2.2 | threadpoolctl | 2.2.0 |
tokenize-rt | 4.2.1 | tornado | 6.4.1 | traitlets | 5.14.3 |
typing_extensions | 4.11.0 | tzdata | 2024.1 | ujson | 5.10.0 |
urllib3 | 2.2.2 | uvicorn | 0.34.0 | wcwidth | 0.2.5 |
wrapt | 1.14.1 | zipp | 3.17.0 | zstandard | 0.23.0 |