Notebook isolation refers to the visibility of variables and classes between notebooks. Databricks supports two types of isolation:
Variable and class isolation
Spark session isolation
Databricks manages user isolation using access modes configured on clusters.
No isolation shared: Multiple users can use the same cluster. Users share credentials set at the cluster level. No data access controls are enforced.
Single user: Only the named user can use the cluster. All commands run with that user’s privileges. Table ACLs in the Hive metastore are not enforced. This access mode supports Unity Catalog.
Shared: Multiple users can use the same cluster. Users are fully isolated from one another, and each user runs commands with their own privileges. Table ACLs in the Hive metastore are enforced. This access mode supports Unity Catalog.
Variables and classes are available only in the current notebook. For example, two notebooks attached to the same cluster can define variables and classes with the same name, but these objects are distinct.
To define a class that is visible to all notebooks attached to the same cluster, define the class in a package cell. Then you can access the class by using its fully qualified name, which is the same as accessing a class in an attached Scala or Java library.
Every notebook attached to a cluster has a pre-defined variable named
spark that represents a
SparkSession is the entry point for using Spark APIs as well as setting runtime configurations.
Spark session isolation is enabled by default. You can also use global temporary views to share temporary views across notebooks. See CREATE VIEW. To disable Spark session isolation, set
true in the Spark configuration.
spark.databricks.session.share true breaks the monitoring used by both streaming notebook cells and streaming jobs. Specifically:
The graphs in streaming cells are not displayed.
Jobs do not block as long as a stream is running (they just finish “successfully”, stopping the stream).
Streams in jobs are not monitored for termination. Instead you must manually call
Calling the Create a new visualization on streaming DataFrames doesn’t work.
Cells that trigger commands in other languages (that is, cells using
%sql) and cells that include other notebooks (that is, cells using
%run) are part of the current notebook. Thus, these cells are in the same session as other notebook cells. By contrast, a notebook workflow runs a notebook with an isolated
SparkSession, which means temporary views defined in such a notebook are not visible in other notebooks.