nanvl

Returns col1 if it is not NaN, or col2 if col1 is NaN. Both inputs should be floating point columns (DoubleType or FloatType). Supports Spark Connect.

For the corresponding Databricks SQL function, see nanvl function.

Syntax

Python

from pyspark.databricks.sql import functions as dbf

dbf.nanvl(col1=<col1>, col2=<col2>)

Parameters

Parameter	Type	Description
col1	pyspark.sql.Column or str	First column to check.
col2	pyspark.sql.Column or str	Second column to return if first is NaN.

Returns

pyspark.sql.Column: value from first column or second if first is NaN .

Examples

Python

from pyspark.databricks.sql import functions as dbf
df = spark.createDataFrame([(1.0, float('nan')), (float('nan'), 2.0)], ("a", "b"))
df.select("*", dbf.nanvl("a", "b"), dbf.nanvl(df.a, df.b)).show()

Output

+---+---+-----------+-----------+
|  a|  b|nanvl(a, b)|nanvl(a, b)|
+---+---+-----------+-----------+
|1.0|NaN|        1.0|        1.0|
|NaN|2.0|        2.0|        2.0|
+---+---+-----------+-----------+