schema_of_csv
Parses a CSV string and infers its schema in DDL format.
Syntax
Python
from pyspark.sql import functions as sf
sf.schema_of_csv(csv, options=None)
Parameters
Parameter | Type | Description |
|---|---|---|
|
| A CSV string or a foldable string column containing a CSV string. |
| dict, optional | Options to control parsing. Accepts the same options as the CSV datasource. |
Returns
pyspark.sql.Column: A string representation of a StructType parsed from the given CSV.
Examples
Example 1: Inferring the schema of a CSV string with different data types
Python
from pyspark.sql import functions as sf
df = spark.range(1)
df.select(sf.schema_of_csv(sf.lit('1|a|true'), {'sep':'|'})).show(truncate=False)
Output
+-------------------------------------------+
|schema_of_csv(1|a|true) |
+-------------------------------------------+
|STRUCT<_c0: INT, _c1: STRING, _c2: BOOLEAN>|
+-------------------------------------------+
Example 2: Inferring the schema of a CSV string with missing values
Python
from pyspark.sql import functions as sf
df = spark.range(1)
df.select(sf.schema_of_csv(sf.lit('1||true'), {'sep':'|'})).show(truncate=False)
Output
+-------------------------------------------+
|schema_of_csv(1||true) |
+-------------------------------------------+
|STRUCT<_c0: INT, _c1: STRING, _c2: BOOLEAN>|
+-------------------------------------------+
Example 3: Inferring the schema of a CSV string with a different delimiter
Python
from pyspark.sql import functions as sf
df = spark.range(1)
df.select(sf.schema_of_csv(sf.lit('1;a;true'), {'sep':';'})).show(truncate=False)
Output
+-------------------------------------------+
|schema_of_csv(1;a;true) |
+-------------------------------------------+
|STRUCT<_c0: INT, _c1: STRING, _c2: BOOLEAN>|
+-------------------------------------------+
Example 4: Inferring the schema of a CSV string with quoted fields
Python
from pyspark.sql import functions as sf
df = spark.range(1)
df.select(sf.schema_of_csv(sf.lit('"1","a","true"'), {'sep':','})).show(truncate=False)
Output
+-------------------------------------------+
|schema_of_csv("1","a","true") |
+-------------------------------------------+
|STRUCT<_c0: INT, _c1: STRING, _c2: BOOLEAN>|
+-------------------------------------------+