format (DataStreamWriter)

Specifies the underlying output data source.

Syntax

format(source)

Parameters

Parameter	Type	Description
source	str	Name of the data source, for example 'parquet' or 'console'.

Parameter	Type	Description
source	str	Name of the data source, for example 'parquet' or 'console'.

Returns

DataStreamWriter

Examples

Python

df = spark.readStream.format("rate").load()
df.writeStream.format("text")
# <...streaming.readwriter.DataStreamWriter object ...>

Write a Rate source stream to CSV:

Python

import tempfile
import time
with tempfile.TemporaryDirectory(prefix="format1") as d:
    with tempfile.TemporaryDirectory(prefix="format2") as cp:
        df = spark.readStream.format("rate").load()
        q = df.writeStream.format("csv").option("checkpointLocation", cp).start(d)
        time.sleep(5)
        q.stop()
        spark.read.schema("timestamp TIMESTAMP, value STRING").csv(d).show()