map_concat
Returns the union of all given maps. For duplicate keys in input maps, the handling is governed by spark.sql.mapKeyDedupPolicy. By default, it throws an exception. If set to LAST_WIN, it uses the last map's value.
Syntax
Python
from pyspark.sql import functions as sf
sf.map_concat(*cols)
Parameters
Parameter | Type | Description |
|---|---|---|
|
| Column names or Column |
Returns
pyspark.sql.Column: A map of merged entries from other maps.
Examples
Example 1: Basic usage of map_concat
Python
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map(3, 'c') as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
Output
+------------------------+
|map_concat(map1, map2) |
+------------------------+
|{1 -> a, 2 -> b, 3 -> c}|
+------------------------+
Example 2: map_concat with three maps
Python
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a') as map1, map(2, 'b') as map2, map(3, 'c') as map3")
df.select(sf.map_concat("map1", "map2", "map3")).show(truncate=False)
Output
+----------------------------+
|map_concat(map1, map2, map3)|
+----------------------------+
|{1 -> a, 2 -> b, 3 -> c} |
+----------------------------+
Example 3: map_concat with empty map
Python
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map() as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
Output
+----------------------+
|map_concat(map1, map2)|
+----------------------+
|{1 -> a, 2 -> b} |
+----------------------+
Example 4: map_concat with null values
Python
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map(3, null) as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
Output
+---------------------------+
|map_concat(map1, map2) |
+---------------------------+
|{1 -> a, 2 -> b, 3 -> NULL}|
+---------------------------+