refreshTable
Invalidates and refreshes all the cached data and metadata of the given table.
Syntax
refreshTable(tableName: str)
Parameters
Parameter | Type | Description |
|---|---|---|
| str | Name of the table to get. Can be qualified with catalog name. |
Examples
Python
# The example below caches a table, and then removes the data.
import tempfile
with tempfile.TemporaryDirectory(prefix="refreshTable") as d:
_ = spark.sql("DROP TABLE IF EXISTS tbl1")
_ = spark.sql(
"CREATE TABLE tbl1 (col STRING) USING TEXT LOCATION '{}'".format(d))
_ = spark.sql("INSERT INTO tbl1 SELECT 'abc'")
spark.catalog.cacheTable("tbl1")
spark.table("tbl1").show()
# +---+
# |col|
# +---+
# |abc|
# +---+
# Because the table is cached, it computes from the cached data as below.
spark.table("tbl1").count()
# 1
# After refreshing the table, it shows 0 because the data does not exist anymore.
spark.catalog.refreshTable("tbl1")
spark.table("tbl1").count()
# 0
# Using the fully qualified name for the table.
spark.catalog.refreshTable("spark_catalog.default.tbl1")
_ = spark.sql("DROP TABLE tbl1")