StreamingQueryManager

Manages all active StreamingQuery instances associated with a SparkSession. Use spark.streams to access this.

Syntax

Python

# Access through SparkSession
spark.streams

Properties

Property	Description
active	Returns a list of all active streaming queries associated with this SparkSession.

Methods

Method	Description
get(id)	Returns an active query by its unique ID.
awaitAnyTermination(timeout)	Waits until any active query terminates, or until the timeout expires.
resetTerminated()	Forgets past terminated queries so that awaitAnyTermination() can be used again to wait for new terminations.
addListener(listener)	Registers a StreamingQueryListener to receive lifecycle event callbacks.
removeListener(listener)	Deregisters a StreamingQueryListener.

Examples

Python

sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('this_query').start()
sqm = spark.streams
[q.name for q in sqm.active]
# ['this_query']
sqm.awaitAnyTermination(5)
# True
sq.stop()
sqm.resetTerminated()