processAllAvailable (StreamingQuery)

Blocks until all available data in the source has been processed and committed to the sink. Intended for testing.

Syntax

processAllAvailable()

Returns

None

Notes

In the case of continually arriving data, this method may block forever. This method is only guaranteed to block until data that has been synchronously appended to a stream source prior to invocation (that is, getOffset must immediately reflect the addition).

Examples

Python

sdf = spark.readStream.format("rate").load()
sq = sdf.writeStream.format('memory').queryName('this_query').start()
sq.processAllAvailable()
sq.stop()