TableValuedFunction.explode

Returns a DataFrame containing a new row for each element in the given array or map. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise.

Syntax

Python

spark.tvf.explode(collection)

Parameters

Parameter	Type	Description
collection	pyspark.sql.Column	Target column to work on.

Returns

pyspark.sql.DataFrame: A DataFrame with a new row for each element.

Examples

Example 1: Exploding an array column

Python

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array(sf.lit(1), sf.lit(2), sf.lit(3))).show()

Output

+---+
|col|
+---+
|  1|
|  2|
|  3|
+---+

Example 2: Exploding a map column

Python

import pyspark.sql.functions as sf
spark.tvf.explode(
    sf.create_map(sf.lit("a"), sf.lit("b"), sf.lit("c"), sf.lit("d"))
).show()

Output

+---+-----+
|key|value|
+---+-----+
|  a|    b|
|  c|    d|
+---+-----+

Example 3: Exploding an array of struct column

Python

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array(
    sf.named_struct(sf.lit("a"), sf.lit(1), sf.lit("b"), sf.lit(2)),
    sf.named_struct(sf.lit("a"), sf.lit(3), sf.lit("b"), sf.lit(4))
)).select("col.*").show()

Output

+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
+---+---+

Example 4: Exploding an empty array column

Python

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array()).show()

Output

+---+
|col|
+---+
+---+

Example 5: Exploding an empty map column

Python

import pyspark.sql.functions as sf
spark.tvf.explode(sf.create_map()).show()

Output

+---+-----+
|key|value|
+---+-----+
+---+-----+