to_variant_object
Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.
Syntax
Python
from pyspark.sql import functions as sf
sf.to_variant_object(col)
Parameters
Parameter | Type | Description |
|---|---|---|
|
| A column with a nested schema or column name. |
Returns
pyspark.sql.Column: a new column of VariantType.
Examples
Example 1: Converting an array containing a nested struct into a variant
Python
from pyspark.sql import functions as sf
from pyspark.sql.types import ArrayType, StructType, StructField, StringType, MapType
schema = StructType([
StructField("i", StringType(), True),
StructField("v", ArrayType(StructType([
StructField("a", MapType(StringType(), StringType()), True)
]), True))
])
data = [("1", [{"a": {"b": 2}}])]
df = spark.createDataFrame(data, schema)
df.select(sf.to_variant_object(df.v))
Output
DataFrame[to_variant_object(v): variant]
Python
df.select(sf.to_variant_object(df.v)).show(truncate=False)
Output
+--------------------+
|to_variant_object(v)|
+--------------------+
|[{"a":{"b":"2"}}] |
+--------------------+