RuntimeConfig

User-facing configuration API, accessible through SparkSession.conf.

Supports Spark Connect

Options set here are automatically propagated to the Hadoop configuration during I/O.

Syntax

Python
spark.conf.set(key, value)
spark.conf.get(key)

Property	Description
`getAll`	Returns all properties set in this configuration as a dict.

Method	Description
`set(key, value)`	Sets the given Spark runtime configuration property. `value` can be a string, integer, or boolean.
`get(key, default)`	Returns the value of the configuration property for the given key. Returns `default` if the key does not exist. Raises an exception if the key is not set and no default is provided.
`unset(key)`	Resets the configuration property for the given key.
`isModifiable(key)`	Returns `True` if the configuration property for the given key is modifiable in the current session.

Set and retrieve a configuration property:

Python
spark.conf.set("key1", "value1")
spark.conf.get("key1")

Output
'value1'

Retrieve a property with a default value:

Python
spark.conf.get("non-existent-key", "my_default")

Output
'my_default'

Unset a configuration property:

Python
spark.conf.set("my_key", "my_value")
spark.conf.unset("my_key")
spark.conf.get("my_key")

Output
pyspark...SparkNoSuchElementException: ... The SQL config "my_key" cannot be found...

Retrieve all configuration properties:

Python
spark.conf.set("key1", "value1")
spark.conf.set("key2", "value2")
spark.conf.getAll

Output
{'key1': 'value1', 'key2': 'value2'}