STREAMING_REAL_TIME_MODE error condition
Streaming real-time mode has the following limitation:
ARBITRARY_STATEFUL_OPERATIONS_NOT_SUPPORTED
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
ASYNC_PROGRESS_TRACKING_CHECKPOINTING_INTERVAL_NON_ZERO
The checkpointing interval for async progress tracking must be set to 0, which means each progress update is checkpointed. Set option asyncProgressTrackingCheckpointIntervalMs to 0 in DataStreamWriter options and retry your query.
ASYNC_PROGRESS_TRACKING_NOT_SUPPORTED
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
BATCH_UNION_NOT_SUPPORTED
Real-time mode does not support union with batch data sources. Ensure that the union streaming query only contains streaming data sources.
CHECKPOINT_FORMAT_V1_NOT_SUPPORTED
Real-time mode does not support checkpoint format v1. If you encountered this while switching to real-time mode, use a fresh checkpoint location.
If you must continue with your existing checkpoint and acknowledge that doing so exposes you to the possibility of data loss on failure, set spark.databricks.streaming.realTimeMode.dangerouslyAllowCheckpointV1.enabled to true.
CLUSTER_CONFIGURATION_NOT_SUPPORTED
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your cluster configuration if present, and then restart your Spark cluster.
DROP_DUPLICATES_WITHIN_WATERMARK_NOT_SUPPORTED
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
EVENT_TIME_BASED_TIMERS_IN_TRANSFORM_WITH_STATE_NOT_SUPPORTED
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
EXACTLY_ONCE_SINK_NOT_SUPPORTED
The sink: <sink> delivers output exactly once, which is not supported by real-time mode. Set the sink option 'mode' to 'atLeastOnce' and retry your query.
IDENTICAL_SOURCES_IN_UNION_NOT_SUPPORTED
Real-time mode does not support union on two or more identical streaming data sources in a single query. This includes scenarios such as referencing the same source DataFrame more than once, or using two data sources with identical configurations for some sources. For Kafka, avoid reusing the same DataFrame and create different ones. For Kinesis, in addition to duplicating data frames, ensure that each DataFrame sets a unique value for the "consumerName" option. Sources provided in the query: <sources>
INPUT_STREAM_NOT_SUPPORTED
The input stream <className> is not supported. See the Real-Time Mode User Guide for a list of supported input streams.
OPERATOR_OR_SINK_NOT_IN_ALLOWLIST
The <errorType>(s): <message> not in the <errorType> allowlist for Real-Time Mode. To bypass this check, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
OUTPUT_MODE_NOT_SUPPORTED
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
SESSION_WINDOWS_NOT_SUPPORTED
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
SINK_NOT_SUPPORTED
The <className> sink is currently not supported. See the Real-Time Mode User Guide for a list of supported sinks.
SQL_CONFIGURATION_NOT_SUPPORTED
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your SparkSession configuration if present, and then restart your Spark cluster.
STATEFUL_OPERATORS_BEFORE_UNION_NOT_SUPPORTED
Streaming queries in real-time mode cannot include stateful operators (for example, aggregate, deduplicate, transformWithState) before a union. Please restructure your query to apply the union before any stateful operations.
STREAM_STREAM_JOIN_NON_INNER_NOT_SUPPORTED
RTM stream-stream join currently only supports Inner join, but got <joinType>.
STREAM_STREAM_JOIN_POLLING_REQUIRED
Stream-stream join in real-time mode requires both input streams to support non-blocking iteration. Enable spark.sql.streaming.realTimeMode.controlMessage.enabled to use this feature.
TRANSFORM_WITH_STATE_IN_PANDAS_NOT_SUPPORTED
transformWithStateInPandas is not supported in Real-Time Mode. Please see the Real-Time Mode User Guide for a list of supported operators.