unpersist

Marks the DataFrame as non-persistent, and remove all blocks for it from memory and disk.

Syntax

unpersist(blocking: bool = False)

Parameters

Parameter	Type	Description
blocking	bool	Whether to block until all blocks are deleted.

Parameter	Type	Description
blocking	bool	Whether to block until all blocks are deleted.

Returns

DataFrame: Unpersisted DataFrame.

Notes

blocking default has changed to False to match Scala in 2.0.

On classic compute, cached data is shared across all Spark sessions on the cluster, so unpersisting it affects all sessions.

On serverless compute, unpersist() marks the cached result as ready to be reclaimed and ignores blocking. After that, the DataFrame and the DataFrames derived from it can no longer be used. See Cache DataFrames on serverless compute.

Examples

Python

df = spark.range(1)
df.persist()
# DataFrame[id: bigint]
df.unpersist()
# DataFrame[id: bigint]
df = spark.range(1)
df.unpersist(True)
# DataFrame[id: bigint]