Skip to main content
Last updated on

Cache DataFrames on serverless compute

Preview

This feature is in Public Preview. Workspace admins can control access to DataFrame caching on serverless compute from the Previews page. See Manage Databricks previews.

On serverless compute, you can cache a DataFrame with df.cache() or df.persist() to reuse an expensive result in your session. Databricks computes the DataFrame when you call cache() and materializes the result as a snapshot. The DataFrame and the DataFrames you build from it then read the snapshot for the rest of your session instead of recomputing it.

Requirements​

  • The DataFrame caching on serverless compute preview is enabled for your workspace (enabled by default). Workspace admins can disable it from the Previews page. The preview isn't available in workspaces with the compliance security profile enabled. See Manage workspace-level previews.
  • Notebooks and jobs on serverless compute must use environment version 5 or above.
  • Databricks Connect for Python must be 18.1.8 or above, and Databricks Connect for Scala must be 18.1.5 or above.

Older clients get the CACHE_REQUIRES_NEWER_CLIENT_ON_SERVERLESS_GC error.

When to use DataFrame caching on serverless compute​

Cache a DataFrame on serverless compute when you need to:

  • Reuse the result of an expensive join, aggregation, or filter several times in the same session.
  • Compute a streaming micro-batch one time in a foreachBatch function, then write it to several sinks.
  • Train or tune several machine learning models on the same prepared feature data.
  • Keep a stable snapshot while you run several steps of an analysis, even if the source tables change.

See Best practices.

Cache DataFrame APIs on serverless compute​

The following example caches one day of orders joined with customer data, computes the day's total revenue, appends the revenue by region to a report table, and then releases the cache. Both results read the cached snapshot instead of running the join again.

Python
from pyspark.sql import functions as F

# The DataFrame to cache: one day of orders joined with customer data.
oct_15_orders = (
    spark.table("main.sales.orders")
    .filter(F.col("order_date") == "2026-10-15")
    .join(spark.table("main.sales.customers"), "customer_id")
)
oct_15_orders.cache()

# Both results are built from oct_15_orders, so they read the cached snapshot.
total_revenue = oct_15_orders.agg(F.sum("amount")).first()[0]
print(f"Total revenue on 2026-10-15: {total_revenue}")

revenue_by_region = oct_15_orders.groupBy("order_date", "region").agg(F.sum("amount").alias("revenue"))
revenue_by_region.write.mode("append").saveAsTable("main.reports.revenue_by_region")

# Release the cache and the DataFrame when you no longer need it.
# oct_15_orders and DataFrames built from it can no longer be used.
oct_15_orders.unpersist()

In the following API descriptions, df stands for any DataFrame in your session, such as oct_15_orders in the preceding example.

df.cache()​

Computes df right away and caches the result as a snapshot for the rest of your session.

  • Returns: The same cached DataFrame, df, now based on the snapshot instead of the computation on its source tables. See the following behavior.
  • Behavior:
    • Runs the DataFrame's query right away and materializes the result as a snapshot before the call returns.
    • After the call, df and the returned DataFrame (df itself) are based on the snapshot. They are no longer connected to the computation on the source tables, so they read the snapshot instead of recomputing the query.
    • DataFrames derived from df, directly or through other derived DataFrames, read the snapshot automatically instead of running the original query again. See Read the snapshot.
    • The snapshot is local to your session. Only the session that cached df can read it. See Isolation and privileges.
    • The snapshot lasts until the session ends, for at most seven days. Databricks then automatically reclaims its storage in the background. Calling df.unpersist() allows it to reclaim the storage earlier. See Lifecycle and time limits and Storage and cleanup.
    • If computing df fails, df.cache() raises the error and df isn't cached.
    • Calling df.cache() on a DataFrame that is already cached does nothing. df isn't computed again.

The snapshot is fixed after you call df.cache(). Data written afterward to the tables that df reads from, and schema changes to those tables such as added or renamed columns, don't affect it. To recompute from the latest data, build the DataFrame again from the source tables, and cache the new DataFrame if you need to.

Read the snapshot​

On serverless compute, cache matching is based on DataFrame derivation. After df.cache(), only the following DataFrames read the snapshot automatically instead of recomputing the query on the source tables:

  • The DataFrame you cached, df.
  • DataFrames derived from df, for example with select(), filter(), groupBy(), or join().
  • DataFrames further chained from those derived DataFrames.

A new DataFrame that repeats the same query reads the source tables, even though its logic is identical, because it isn't derived from df.

This makes cache matching explicit and under your control: a DataFrame reads cached data only when you derive it from a cached DataFrame. Every other DataFrame reads the source tables.

Python
oct_15_orders = (
    spark.table("main.sales.orders")
    .filter(F.col("order_date") == "2026-10-15")
    .join(spark.table("main.sales.customers"), "customer_id")
)
# The join runs one time, when you call cache().
oct_15_orders.cache()

# Derived from oct_15_orders: read the snapshot, so the join doesn't run again.
large_orders = oct_15_orders.filter(F.col("amount") > 1000)
revenue_by_region = oct_15_orders.groupBy("order_date", "region").agg(F.sum("amount").alias("revenue"))

# Derived from large_orders, which is derived from oct_15_orders: also reads the snapshot.
large_orders_by_region = large_orders.groupBy("region").count()

# Not derived from oct_15_orders: reads the source tables and runs the join again.
same_query = (
    spark.table("main.sales.orders")
    .filter(F.col("order_date") == "2026-10-15")
    .join(spark.table("main.sales.customers"), "customer_id")
)

Because df reads the snapshot instead of the DataFrames it was built from, you can't reference the columns of df through those DataFrames. See Limitations.

df.persist(storageLevel)​

An alias of df.cache() that accepts a storage level.

  • Parameters:
    • storageLevel (optional): A StorageLevel, such as StorageLevel.DISK_ONLY. serverless compute accepts any storage level and ignores it. The snapshot is stored the same way for every storage level, and df.storageLevel returns StorageLevel.DISK_ONLY.

The return value and behavior are the same as df.cache().

df.unpersist(blocking)​

Releases the cached snapshot of df and destroys the DataFrame.

  • Parameters:
    • blocking (optional): Ignored. df.unpersist() returns right away for either value.
  • Returns: The same DataFrame, df, now destroyed.
  • Behavior:
    • Marks the snapshot as ready to be reclaimed. Databricks then deletes the cached data and reclaims the storage in the background.
    • Destroys df: df and any DataFrames previously derived from it are no longer usable, and any subsequent query on them raises CACHE_DATAFRAME_ACCESS_AFTER_UNCACHE. If you cache a DataFrame derived from df before you call df.unpersist(), that DataFrame stays usable, because it has its own snapshot.
    • To continue working with the same data source, build a new DataFrame from the original data source, instead of reusing the uncached DataFrame.
    • Calling df.unpersist() on a DataFrame that was never cached, or that has already been uncached, is a no-op.

Call df.unpersist() when you no longer need the cached data, for example when no later step in your notebook or job reads it. This allows Databricks to reclaim the storage earlier. Don't use df.unpersist() to refresh data. To work with the latest data, build a new DataFrame from the source tables.

Python
oct_15_orders = spark.table("main.sales.orders").filter(F.col("order_date") == "2026-10-15")
oct_15_orders.cache()
large_orders = oct_15_orders.filter(F.col("amount") > 1000)
orders_per_customer = oct_15_orders.groupBy("customer_id").count()
orders_per_customer.cache()  # Has its own snapshot.

# Acknowledge that the cached data is no longer needed.
# This releases the storage and destroys oct_15_orders.
oct_15_orders.unpersist()

oct_15_orders.count()  # Raises CACHE_DATAFRAME_ACCESS_AFTER_UNCACHE.
large_orders.count()  # Raises CACHE_DATAFRAME_ACCESS_AFTER_UNCACHE.
orders_per_customer.count()  # Reads its own snapshot.

# If you need the data again later, build a new DataFrame from the source tables.
oct_15_orders = spark.table("main.sales.orders").filter(F.col("order_date") == "2026-10-15")

df.storageLevel​

Gets the storage level of df.

  • Returns: StorageLevel.DISK_ONLY while df is cached. StorageLevel.NONE before you cache it and after you call df.unpersist().
Python
from pyspark import StorageLevel

oct_15_orders.cache()
oct_15_orders.storageLevel == StorageLevel.DISK_ONLY  # True

For the method signatures, see the PySpark reference for cache, persist, unpersist, and storageLevel.

Lifecycle and time limits​

Cached data exists only within the Databricks session where you cache it. It becomes inaccessible when the session ends, and it lasts at most seven days. After the seven-day limit, using a cached DataFrame raises an error instead of reading the source tables again. To cache again, start a new session.

Storage and cleanup​

Databricks manages the storage for cached data automatically. Cached data is stored in cloud storage, with the specific storage location depending on your workspace type:

After cached data becomes inaccessible, Databricks automatically reclaims the storage in the background. This includes cases where the session ends because of a disconnection or a crash. Calling df.unpersist() allows Databricks to reclaim the storage earlier.

Isolation and privileges​

Any user who can run code on serverless compute can cache a DataFrame. You don't need any extra privileges.

Cached data provides session-level isolation. Only the session that cached a DataFrame can read its cached data. Other users can't access it, including other users of the same notebook.

The cached data contains what you were allowed to read when you cached it. Permission changes, row filters, and column masks applied to the source tables afterward don't apply to the cached data.

Best practices​

  • Cache when you need a stable snapshot: If you run several transformations, aggregations, or drill-downs on the same computed data, the snapshot gives them a consistent baseline that doesn't change when the source tables change.
  • Cache expensive computations, not plain table reads: df.cache() materializes the query result, and later reads scan that snapshot. If you cache a plain table read, such as spark.table("main.sales.orders").cache(), later reads still scan a full copy of the table, so caching saves little. Caching pays off after expensive work such as joins, aggregations, and complex filters.
  • Don't cache data that you read only one time: df.cache() computes and stores the result right away, which takes time and storage. If you read the result only one time, the upfront cost isn't worth it.
  • Use the cache by deriving from the cached DataFrame: To read the snapshot, build later operations from the cached DataFrame variable, for example df.select(...) or df.filter(...). An independent DataFrame built from the same computation on the source tables doesn't use the cache.
  • Don't use cache for cross-session or cross-user data sharing: Cached data is private to your session. Other sessions, notebooks, job runs, and users can't read it. To share computed results across sessions or with other users, write them to a permanent table.
  • Keep sessions with cached data under seven days: See Lifecycle and time limits. If your workflow needs the data for longer, write it to a table.
  • Call df.unpersist() when you no longer need the cached data: This allows Databricks to reclaim the storage earlier. After the call, the DataFrame and the DataFrames derived from it can't be used.
  • Don't use df.unpersist() to refresh data: df.unpersist() ends the snapshot's life. To pick up changes in the source tables, build a new DataFrame from the source tables.

Limitations​

DataFrame caching on serverless compute has the following limitations:

  • Column names and types: The following DataFrames can't be cached. Rename or cast the columns before you cache, for example with .alias("total_amount").
    • DataFrames with duplicate column names.
    • DataFrames with column names that contain spaces or any of the characters ,;{}()=, such as sum(amount) from an aggregation.
    • DataFrames with interval columns.
    • DataFrames with arrays, maps, or structs that contain NULL-typed values.
    • DataFrames with no columns, or with only NULL-typed columns.
  • Column references: After you cache a DataFrame, you can't reference its columns through the DataFrames it was built from. For example, after oct_15_orders = orders.join(customers, "customer_id") and oct_15_orders.cache(), oct_15_orders.select(orders.amount) raises CANNOT_RESOLVE_DATAFRAME_COLUMN. Use oct_15_orders.amount or "amount" instead.
  • SQL and catalog cache APIs: SQL cache statements, such as CACHE TABLE, and the table cache methods of spark.catalog aren't supported on serverless compute. See Caching limitations. To reuse a query result in SQL, use a session-local temporary table.
  • Information schema queries: When many sessions in a metastore cache DataFrames, queries on information_schema.volumes and information_schema.schemata can be slow or time out. Filtering volumes by catalog doesn't make the query faster. Queries on information_schema.tables aren't affected.
  • AWS GovCloud: DataFrame caching isn't supported on serverless compute in AWS GovCloud or AWS GovCloud DoD. See Unavailable features.

Migrate from classic compute​

For most workloads, caching on serverless compute gives the same results as on classic compute, and existing code runs without changes. The following table lists the differences.

Behavior

Serverless compute

Classic compute

Cache matching

Only the cached DataFrame and the DataFrames derived from it automatically use the cache.

Any query with the same plan shape automatically uses the cache.

Data freshness

A consistent snapshot taken at the time you call df.cache().

Varies: cached data is refreshed in certain cases and serves a snapshot in others.

Sharing scope

Private to your session.

Shared across sessions within the cluster.

Materialization

Eager: computed right away when you call df.cache().

Lazy: computed at the first action on the DataFrame.

df.unpersist()

Destroys the DataFrame and releases its snapshot deterministically.

Releases the in-memory cache. Later use of the DataFrame may get the latest or stale data, depending on the product and data source.

Behavior

Serverless compute

Classic compute

Cache matching

Only the cached DataFrame and the DataFrames derived from it automatically use the cache.

Any query with the same plan shape automatically uses the cache.

Data freshness

A consistent snapshot taken at the time you call df.cache().

Varies: cached data is refreshed in certain cases and serves a snapshot in others.

Sharing scope

Private to your session.

Shared across sessions within the cluster.

Materialization

Eager: computed right away when you call df.cache().

Lazy: computed at the first action on the DataFrame.

df.unpersist()

Destroys the DataFrame and releases its snapshot deterministically.

Releases the in-memory cache. Later use of the DataFrame may get the latest or stale data, depending on the product and data source.

The following describes the first two differences in more detail:

  • Cache matching:
    • Workloads affected by the change: Most workloads aren't affected. If you keep using the cached DataFrame variable, for example with df.select(...) or df.filter(...), the cache works as before. The change affects only workloads where an independently built DataFrame or SQL query reuses the cache because it has the same plan shape. On serverless compute, those queries read the source tables.
    • If your workload is affected: Use the cached DataFrame variable wherever you need the cached result, instead of building the same query again.
  • Data freshness:
    • Workloads affected by the change: Most workloads aren't affected, because reading cached data again after writing to its source tables is uncommon. The change affects only workloads that write to the source tables and then expect the cached DataFrame to show the new data.
    • If your workload is affected: After you write to the source tables, build a new DataFrame from them instead of reusing the cached one.

Additional resources​

On this page