IPython kernel

The IPython kernel is a Jupyter kernel for Python code execution. Jupyter, and other compatible notebooks, use the IPython kernel for executing Python notebook code.

In Databricks Runtime 10.1 and above, you can configure a Databricks cluster to use the IPython kernel to execute Python code.

Benefits of using the IPython kernel

The IPython kernel allows Databricks to add better support for open source tools built for Jupyter notebooks. Using the IPython kernel on Databricks adds support for IPython’s display and output tooling. See IPython.core.display for more information. Also, the IPython kernel captures the stdout and stderr outputs of child processes created by a notebook, allowing that output to be included in the notebook’s command results.

How to use the IPython kernel with Databricks

When you start a Databricks cluster with Databricks Runtime 10.1 or above, you can configure the cluster to use the IPython kernel for Python code execution. To do this, set the Spark configuration spark.databricks.python.defaultPythonRepl ipykernel.

Known issues

There are several known limitations when using the IPython kernel with Databricks notebooks.

  • Unlike the default Databricks Python notebooks, when you use the IPython kernel, the notebook does not automatically display the schema when creating DataFrames.
  • The IPython kernel has more limited autocomplete functionality. Autocomplete cannot be used when code is executing.
  • SparkContext.setJobGroup does not work with the IPython kernel.
  • The IPython command update_display only updates the outputs of the current cell.