sum_distinct
Aggregate function: returns the sum of distinct values in the expression.
Syntax
Python
from pyspark.sql import functions as sf
sf.sum_distinct(col)
Parameters
Parameter | Type | Description |
|---|---|---|
|
| Target column to compute on. |
Returns
pyspark.sql.Column: the column for computed results.
Examples
Example 1: Using sum_distinct function on a column with all distinct values
Python
from pyspark.sql import functions as sf
df = spark.createDataFrame([(1,), (2,), (3,), (4,)], ["numbers"])
df.select(sf.sum_distinct('numbers')).show()
Output
+---------------------+
|sum(DISTINCT numbers)|
+---------------------+
| 10|
+---------------------+
Example 2: Using sum_distinct function on a column with no distinct values
Python
from pyspark.sql import functions as sf
df = spark.createDataFrame([(1,), (1,), (1,), (1,)], ["numbers"])
df.select(sf.sum_distinct('numbers')).show()
Output
+---------------------+
|sum(DISTINCT numbers)|
+---------------------+
| 1|
+---------------------+
Example 3: Using sum_distinct function on a column with null and duplicate values
Python
from pyspark.sql import functions as sf
df = spark.createDataFrame([(None,), (1,), (1,), (2,)], ["numbers"])
df.select(sf.sum_distinct('numbers')).show()
Output
+---------------------+
|sum(DISTINCT numbers)|
+---------------------+
| 3|
+---------------------+