dropFields

Remove fields from a struct column.

Syntax

Python

dropFields(*fieldNames)

Parameters

Parameter	Type	Description
fieldNames	str	One or more field names to drop

Parameter	Type	Description
fieldNames	str	One or more field names to drop

Returns

Column

Examples

Python

from pyspark.sql import Row
from pyspark.sql.functions import col, lit
df = spark.createDataFrame([
    Row(a=Row(b=1, c=2, d=3, e=Row(f=4, g=5, h=6)))])
df.withColumn('a', df['a'].dropFields('b')).show()

Output

# +-----------------+
# |                a|
# +-----------------+
# |{2, 3, {4, 5, 6}}|
# +-----------------+

Python

df.withColumn('a', df['a'].dropFields('b', 'c')).show()

Output

# +--------------+
# |             a|
# +--------------+
# |{3, {4, 5, 6}}|
# +--------------+

Dropping multiple nested fields directly:

Python

df.withColumn("a", col("a").dropFields("e.g", "e.h")).show()

Output

# +--------------+
# |             a|
# +--------------+
# |{1, 2, 3, {4}}|
# +--------------+

Python

df.select(col("a").withField(
    "e", col("a.e").dropFields("g", "h")).alias("a")
).show()

Output

# +--------------+
# |             a|
# +--------------+
# |{1, 2, 3, {4}}|
# +--------------+