drop (DataFrame)

Returns a new DataFrame without specified columns. This is a no-op if the schema doesn't contain the given column name(s).

Syntax

drop(*cols: "ColumnOrName")

Parameters

Parameter	Type	Description
cols	str or Column	A name of the column, or the Column to be dropped.

Returns

DataFrame: A new DataFrame without the specified columns.

Notes

When an input is a column name, it is treated literally without further interpretation. Otherwise, it will try to match the equivalent expression. So dropping a column by its name drop(colName) has a different semantic with directly dropping the column drop(col(colName)).

Examples

Python

df = spark.createDataFrame(
    [(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"])
df.drop('age').show()
# +-----+
# | name|
# +-----+
# |  Tom|
# |Alice|
# |  Bob|
# +-----+

df.drop(df.age).show()
# +-----+
# | name|
# +-----+
# |  Tom|
# |Alice|
# |  Bob|
# +-----+

df2 = spark.createDataFrame([(80, "Tom"), (85, "Bob")], ["height", "name"])
df.join(df2, df.name == df2.name).drop('name').sort('age').show()
# +---+------+
# |age|height|
# +---+------+
# | 14|    80|
# | 16|    85|
# +---+------+