localCheckpoint

Returns a locally checkpointed version of this DataFrame. Checkpointing can be used to truncate the logical plan of this DataFrame, which is especially useful in iterative algorithms where the plan may grow exponentially. Local checkpoints are stored in the executors using the caching subsystem and therefore they are not reliable.

Syntax

localCheckpoint(eager: bool = True, storageLevel: Optional[StorageLevel] = None)

Parameters

Parameter	Type	Description
eager	bool, optional, default True	Whether to checkpoint this DataFrame immediately.
storageLevel	StorageLevel, optional, default None	The StorageLevel with which the checkpoint will be stored. If not specified, default for RDD local checkpoints.

Returns

DataFrame: Checkpointed DataFrame.

Notes

This API is experimental.

Examples

Python

df = spark.createDataFrame([
    (14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
df.localCheckpoint(False)
# DataFrame[age: bigint, name: string]