This article covers Databricks Connect for Databricks Runtime 13.0 and above.
This article provides troubleshooting information for Databricks Connect for Python. Databricks Connect enables you to connect popular IDEs, notebook servers, and custom applications to Databricks clusters. See What is Databricks Connect?. For the Scala version of this article, see Troubleshooting Databricks Connect for Scala.
Check the Python version you are using locally has at least the same minor release as the version on the cluster (for example,
3.10.10 is OK,
3.9 is not).
If you have multiple Python versions installed locally, ensure that Databricks Connect is using the right one by setting the
PYSPARK_PYTHON environment variable (for example,
databricks-connect package conflicts with PySpark. Having both installed will cause errors when initializing the Spark context in Python. This can manifest in several ways, including “stream corrupted” or “class not found” errors. If you have PySpark installed in your Python environment, ensure it is uninstalled before installing databricks-connect. After uninstalling PySpark, make sure to fully re-install the Databricks Connect package:
pip3 uninstall pyspark pip3 uninstall databricks-connect pip3 install --upgrade "databricks-connect==14.0.*" # or X.Y.* to match your specific cluster version.
It is possible your PATH is configured so that commands like
spark-shell will be running some other previously installed binary instead of the one provided with Databricks Connect. You should make sure either the Databricks Connect binaries take precedence, or remove the previously installed ones.
If you can’t run commands like
spark-shell, it is also possible your PATH was not automatically set up by
pip3 install and you’ll need to add the installation
bin dir to your PATH manually. It’s possible to use Databricks Connect with IDEs even if this isn’t set up.
If you are using Databricks Connect on Windows and see:
The filename, directory name, or volume label syntax is incorrect.
Databricks Connect was installed into a directory with a space in your path. You can work around this by either installing into a directory path without spaces, or configuring your path using the short name form.