dropna

Returns a new DataFrame omitting rows with null or NaN values. DataFrame.dropna and DataFrameNaFunctions.drop are aliases of each other.

Syntax

dropna(how: str = "any", thresh: Optional[int] = None, subset: Optional[Union[str, Tuple[str, ...], List[str]]] = None)

Parameters

Parameter	Type	Description
how	str, optional, default 'any'	the values that can be 'any' or 'all'. If 'any', drop a row if it contains any nulls. If 'all', drop a row only if all its values are null.
thresh	int, optional, default None	If specified, drop rows that have less than thresh non-null values. This overwrites the how parameter.
subset	str, tuple or list, optional	optional list of column names to consider.

Parameter	Type	Description
how	str, optional, default 'any'	the values that can be 'any' or 'all'. If 'any', drop a row if it contains any nulls. If 'all', drop a row only if all its values are null.
thresh	int, optional, default None	If specified, drop rows that have less than thresh non-null values. This overwrites the how parameter.
subset	str, tuple or list, optional	optional list of column names to consider.

Returns

DataFrame: DataFrame with null only rows excluded.

Examples

Python

from pyspark.sql import Row
df = spark.createDataFrame([
    Row(age=10, height=80.0, name="Alice"),
    Row(age=5, height=float("nan"), name="Bob"),
    Row(age=None, height=None, name="Tom"),
    Row(age=None, height=float("nan"), name=None),
])

df.na.drop().show()
# +---+------+-----+
# |age|height| name|
# +---+------+-----+
# | 10|  80.0|Alice|
# +---+------+-----+

df.na.drop(how='all').show()
# +----+------+-----+
# | age|height| name|
# +----+------+-----+
# |  10|  80.0|Alice|
# |   5|   NaN|  Bob|
# |NULL|  NULL|  Tom|
# +----+------+-----+

df.na.drop(thresh=2).show()
# +---+------+-----+
# |age|height| name|
# +---+------+-----+
# | 10|  80.0|Alice|
# |  5|   NaN|  Bob|
# +---+------+-----+