SparkSession

The entry point to programming Spark with the Dataset and DataFrame API. A SparkSession can be used to create DataFrames, register DataFrames as tables, execute SQL over tables, cache tables, and read parquet files.

Syntax

Python
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

Properties

Property	Description
`builder`	Interface for building the session configuration.
`catalog`	Interface through which the user may create, drop, alter or query underlying databases, tables, functions, etc.
`client`	Gives access to the Spark Connect client. Spark Connect only.
`conf`	Runtime configuration interface for Spark.
`dataSource`	Returns a DataSourceRegistration for data source registration.
`profile`	Returns a Profile for performance/memory profiling.
`read`	Returns a DataFrameReader that can be used to read data as a DataFrame.
`readStream`	Returns a DataStreamReader that can be used to read data streams as a streaming DataFrame.
`sparkContext`	Returns the underlying SparkContext. Classic mode only.
`streams`	Returns a StreamingQueryManager that allows managing all active streaming queries.
`tvf`	Returns a TableValuedFunction for calling table-valued functions (TVFs).
`udf`	Returns a UDFRegistration for UDF registration.
`udtf`	Returns a UDTFRegistration for UDTF registration.
`version`	The version of Spark on which this application is running.

Property	Description
`builder`	Interface for building the session configuration.
`catalog`	Interface through which the user may create, drop, alter or query underlying databases, tables, functions, etc.
`client`	Gives access to the Spark Connect client. Spark Connect only.
`conf`	Runtime configuration interface for Spark.
`dataSource`	Returns a DataSourceRegistration for data source registration.
`profile`	Returns a Profile for performance/memory profiling.
`read`	Returns a DataFrameReader that can be used to read data as a DataFrame.
`readStream`	Returns a DataStreamReader that can be used to read data streams as a streaming DataFrame.
`sparkContext`	Returns the underlying SparkContext. Classic mode only.
`streams`	Returns a StreamingQueryManager that allows managing all active streaming queries.
`tvf`	Returns a TableValuedFunction for calling table-valued functions (TVFs).
`udf`	Returns a UDFRegistration for UDF registration.
`udtf`	Returns a UDTFRegistration for UDTF registration.
`version`	The version of Spark on which this application is running.

Methods

Method	Description
`createDataFrame(data, schema, samplingRatio, verifySchema)`	Creates a DataFrame from an RDD, a list, a pandas DataFrame, a numpy ndarray, or a pyarrow Table.
`sql(sqlQuery, args, **kwargs)`	Returns a DataFrame representing the result of the given query.
`table(tableName)`	Returns the specified table as a DataFrame.
`range(start, end, step, numPartitions)`	Creates a DataFrame with a single LongType column named `id`, containing elements in a range.
`newSession()`	Returns a new SparkSession with separate SQLConf, registered temporary views, and UDFs, but shared SparkContext and table cache. Classic mode only.
`getActiveSession()`	Returns the active SparkSession for the current thread.
`active()`	Returns the active or default SparkSession for the current thread.
`stop()`	Stops the underlying SparkContext.
`addArtifacts(*path, pyfile, archive, file)`	Adds artifact(s) to the client session.
`interruptAll()`	Interrupts all operations of this session currently running on the server.
`interruptTag(tag)`	Interrupts all operations of this session with the given tag.
`interruptOperation(op_id)`	Interrupts an operation of this session with the given operationId.
`addTag(tag)`	Adds a tag to be assigned to all operations started by this thread in this session.
`removeTag(tag)`	Removes a tag previously added for operations started by this thread.
`getTags()`	Gets the tags currently set to be assigned to all operations started by this thread.
`clearTags()`	Clears the current thread's operation tags.

Method	Description
`createDataFrame(data, schema, samplingRatio, verifySchema)`	Creates a DataFrame from an RDD, a list, a pandas DataFrame, a numpy ndarray, or a pyarrow Table.
`sql(sqlQuery, args, **kwargs)`	Returns a DataFrame representing the result of the given query.
`table(tableName)`	Returns the specified table as a DataFrame.
`range(start, end, step, numPartitions)`	Creates a DataFrame with a single LongType column named `id`, containing elements in a range.
`newSession()`	Returns a new SparkSession with separate SQLConf, registered temporary views, and UDFs, but shared SparkContext and table cache. Classic mode only.
`getActiveSession()`	Returns the active SparkSession for the current thread.
`active()`	Returns the active or default SparkSession for the current thread.
`stop()`	Stops the underlying SparkContext.
`addArtifacts(*path, pyfile, archive, file)`	Adds artifact(s) to the client session.
`interruptAll()`	Interrupts all operations of this session currently running on the server.
`interruptTag(tag)`	Interrupts all operations of this session with the given tag.
`interruptOperation(op_id)`	Interrupts an operation of this session with the given operationId.
`addTag(tag)`	Adds a tag to be assigned to all operations started by this thread in this session.
`removeTag(tag)`	Removes a tag previously added for operations started by this thread.
`getTags()`	Gets the tags currently set to be assigned to all operations started by this thread.
`clearTags()`	Clears the current thread's operation tags.

Syntax​

Properties​

Methods​

Syntax

Properties

Methods