eqNullSafe

Equality test that is safe for null values.

Added in Databricks Runtime 11.0

Changed in Databricks Runtime 13.0: Supports Spark Connect.

Syntax

Python

eqNullSafe(other)

Parameters

Parameter	Type	Description
other	Column or value	A value or Column to compare

Parameter	Type	Description
other	Column or value	A value or Column to compare

Returns

Column

Notes

Unlike Pandas, PySpark doesn't consider NaN values to be NULL. See the NaN Semantics for details.

Examples

Python

from pyspark.sql import Row
df1 = spark.createDataFrame([
    Row(id=1, value='foo'),
    Row(id=2, value=None)
])
df1.select(
    df1['value'] == 'foo',
    df1['value'].eqNullSafe('foo'),
    df1['value'].eqNullSafe(None)
).show()

Output

# +-------------+---------------+----------------+
# |(value = foo)|(value <=> foo)|(value <=> NULL)|
# +-------------+---------------+----------------+
# |         true|           true|           false|
# |         NULL|          false|            true|
# +-------------+---------------+----------------+