builder

Interface for building the session configuration.

Syntax

builder

Methods

Method	Description
appName(name)	Sets a name for the application, which is shown in the Spark web UI.
create()	Creates a new SparkSession.
config(key, value, conf, map)	Sets a config option. Options are automatically propagated to both SparkConf and SparkSession's own configuration.
enableHiveSupport()	Enables Hive support, including connectivity to a persistent Hive metastore.
getOrCreate()	Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.
master(master)	Sets the Spark master URL to connect to.
remote(url)	Sets the Spark remote URL to connect via Spark Connect.

Method	Description
appName(name)	Sets a name for the application, which is shown in the Spark web UI.
create()	Creates a new SparkSession.
config(key, value, conf, map)	Sets a config option. Options are automatically propagated to both SparkConf and SparkSession's own configuration.
enableHiveSupport()	Enables Hive support, including connectivity to a persistent Hive metastore.
getOrCreate()	Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.
master(master)	Sets the Spark master URL to connect to.
remote(url)	Sets the Spark remote URL to connect via Spark Connect.

Examples

Python

spark = (
    SparkSession.builder
        .master("local")
        .appName("Word Count")
        .config("spark.some.config.option", "some-value")
        .getOrCreate()
)

Python

spark.sql("SELECT * FROM range(10) where id > 7").show()

Output

+---+
| id|
+---+
|  8|
|  9|
+---+

Python

spark.createDataFrame([('Alice', 1)], ['name', 'age']).show()

Output

+-----+---+
| name|age|
+-----+---+
|Alice|  1|
+-----+---+

Python

spark.range(1, 7, 2).show()

Output

+---+
| id|
+---+
|  1|
|  3|
|  5|
+---+