checkpoint

Returns a checkpointed version of this DataFrame. Checkpointing can be used to truncate the logical plan of this DataFrame, which is especially useful in iterative algorithms where the plan may grow exponentially. It will be saved to files inside the checkpoint directory set with SparkContext.setCheckpointDir, or spark.checkpoint.dir configuration.

Syntax

checkpoint(eager: bool = True)

Parameters

Parameter	Type	Description
eager	bool, optional, default True	Whether to checkpoint this DataFrame immediately.

Parameter	Type	Description
eager	bool, optional, default True	Whether to checkpoint this DataFrame immediately.

Returns

DataFrame: Checkpointed DataFrame.

Notes

This API is experimental.

Examples

Serverless compatibility

Databricks recommends moving away from DataFrame.checkpoint() as it is not compatible with Databricks serverless compute architecture. Write the DataFrame to a Delta table instead.

Python

df = spark.createDataFrame([
    (14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
df.checkpoint(False)
# DataFrame[age: bigint, name: string]