Real-time mode in Structured Streaming
This page describes the concepts behind real-time mode in Structured Streaming, including what it is, how it achieves low latency, and when to use it.
What is real-time mode?
Real-time mode is a trigger type for Structured Streaming that enables ultra-low latency data processing with end-to-end latency as low as five milliseconds. Use real-time mode for operational workloads that require immediate response to streaming data, such as fraud detection, real-time personalization, and instant decision-making systems.
How real-time mode achieves low latency
Real-time mode improves the execution architecture by:
- Executing long-running batches (the default is five minutes), in which the system processes data as it becomes available in the source.
- Scheduling all stages of the query simultaneously. This requires the number of available task slots to be equal to or greater than the number of tasks of all the stages in a batch.
- Passing data between stages as soon as it is produced using a streaming shuffle.
At the end of processing a batch, and before the next batch starts, Structured Streaming checkpoints progress and publishes metrics. The batch duration affects checkpointing frequency:
- For longer batches, checkpointing occurs less frequently, which means longer replays on failure and delayed metrics availability.
- For shorter batches, checkpointing occurs more frequently, which may affect latency.
Databricks recommends benchmarking real-time mode against your target workload to find the appropriate trigger interval.
When to use real-time mode
Select real-time mode when your use case requires:
- Sub-second latency: Applications that need to respond to data within milliseconds. For example, blocking or flagging a credit card transaction in real time if a fraud score exceeds a threshold based on unusual location, large transaction size, or rapid spending patterns.
- Operational decision-making: Systems that trigger immediate actions based on incoming data. For example, delivering a promotional message when clickstream data shows a user has been browsing for a product, offering a discount if they purchase within the next 15 minutes.
- Continuous processing: Workloads where data must be processed as soon as it arrives, rather than in periodic batches.
Use micro-batch mode (the default Structured Streaming trigger) when your use case requires:
- Analytical processing: ETL pipelines, data transformations, and medallion architecture implementations where latency requirements are measured in seconds or minutes.
- Cost optimization: Workloads where sub-second latency is not required, as real-time mode requires dedicated compute resources.
- The checkpoint frequency matters: Applications that benefit from more frequent checkpointing for faster recovery.
Feature support and limitations
For a complete list of supported environments, languages, compute types, sources, sinks, operators, and known limitations, see Real-time mode reference.