trigger (DataStreamWriter)

Sets the trigger for the streaming query. If not set, the query runs as fast as possible, equivalent to processingTime='0 seconds'. Only one trigger parameter can be set at a time.

See Configure Structured Streaming trigger intervals.

Compute and pipeline execution

In Lakeflow pipelines, the pipeline manages stream execution. Use pipeline execution settings to control when pipeline data is processed.

When you use DataStreamWriter directly on serverless compute for notebooks and jobs, use availableNow=True. The default trigger, processingTime, continuous, and realTime are not supported on that compute. These restrictions do not apply to serverless pipeline execution modes. See Streaming limitations.

Syntax

trigger(*, processingTime=None, once=None, continuous=None, availableNow=None, realTime=None)

Parameters

Parameter	Type	Description
processingTime	str, optional	A processing time interval string (for example, '5 seconds', '1 minute'). Runs a microbatch query periodically based on the processing time.
once	bool, optional	If True, processes only one batch of data then terminates the query. Deprecated in Databricks Runtime 11.3 LTS and above. Use availableNow=True instead.
continuous	str, optional	Not supported on Databricks. This parameter enables Apache Spark's experimental Continuous Processing mode, which differs from continuous pipeline mode. Use real-time mode for low-latency Structured Streaming queries.
availableNow	bool, optional	If True, processes all available data in multiple batches then terminates the query.
realTime	str, optional	A batch duration string (for example, '5 minutes'). Processes records as they arrive within a long-running batch. The value controls batch duration, not record-processing latency. See Real-time mode in Structured Streaming.

Parameter	Type	Description
processingTime	str, optional	A processing time interval string (for example, '5 seconds', '1 minute'). Runs a microbatch query periodically based on the processing time.
once	bool, optional	If True, processes only one batch of data then terminates the query. Deprecated in Databricks Runtime 11.3 LTS and above. Use availableNow=True instead.
continuous	str, optional	Not supported on Databricks. This parameter enables Apache Spark's experimental Continuous Processing mode, which differs from continuous pipeline mode. Use real-time mode for low-latency Structured Streaming queries.
availableNow	bool, optional	If True, processes all available data in multiple batches then terminates the query.
realTime	str, optional	A batch duration string (for example, '5 minutes'). Processes records as they arrive within a long-running batch. The value controls batch duration, not record-processing latency. See Real-time mode in Structured Streaming.

Returns

DataStreamWriter

Examples

Python

df = spark.readStream.format("rate").load()

Trigger execution every 5 seconds:

Python

df.writeStream.trigger(processingTime='5 seconds')
# <...streaming.readwriter.DataStreamWriter object ...>

Process all available data in multiple batches:

Python

df.writeStream.trigger(availableNow=True)
# <...streaming.readwriter.DataStreamWriter object ...>

Configure real-time mode with five-minute batches. Records are processed as they arrive during each batch. Use a supported source, sink, and compute type:

Python

kafka_df = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "<bootstrap-servers>")
    .option("subscribe", "<input-topic>")
    .load())
kafka_df.writeStream.trigger(realTime="5 minutes")
# <...streaming.readwriter.DataStreamWriter object ...>